Virtual reality man-machine interaction method

Abstract

The invention relates to the technical field of man-machine interaction, in particular to a virtual reality man-machine interaction method. The method comprises the following steps: S1, designing an interactive interface conforming to a virtual environment; s2, collecting hand actions of a user through a camera; s3, performing real-time processing and analysis on the hand action of the user through a key point detection OpenPose model, and identifying the intention of the hand action of the user; and S4, adjusting the state of the virtual environment according to the intention of the hand action of the user. According to the invention, by combining the YOLO model and the OpenPose model, accurate recognition and real-time response to the hand action of the user are realized. Static features (such as joint angles and palm normals) and dynamic features (such as motion trails, speeds and accelerated speeds) of hand actions are comprehensively utilized, so that visual and natural interaction experience is provided. The user can complete operations such as'grabbing 'and'pointing' through visual gestures, additional equipment is not needed, and the immersion and interactivity of virtual reality are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a virtual reality human-computer interaction method. Background Art

[0002] Traditional human-computer interaction methods (such as mouse, keyboard, game controller, etc.) lack naturalness and immersion in virtual reality environments, and cannot fully simulate the user's intuitive movements in reality, affecting the real experience of virtual reality; hand action recognition needs to deal with real-time, precision, and robustness issues, such as how to separate the hand area from the complex environment background, how to deal with noise in action detection, and how to accurately analyze static gestures and dynamic trajectories, which puts high performance requirements on models and algorithms. Therefore, a virtual reality human-computer interaction method is provided. Summary of the invention

[0003] The purpose of the present invention is to provide a virtual reality human-computer interaction method to solve the limitations of the traditional interaction method and the problem of low accuracy in recognizing hand movements proposed in the above background technology.

[0004] To achieve the above object, the present invention provides a virtual reality human-computer interaction method, comprising the following steps: S1. Design an interactive interface that fits the virtual environment; S2, collecting user hand movements through a camera; S3, through the key point detection OpenPose model to process and analyze the user's hand movements in real time and identify the intention of the user's hand movements; S4. Adjust the state of the virtual environment according to the intention of the user's hand movement.

[0005] As a further improvement of the technical solution, in S1, designing an interactive interface that conforms to the virtual environment includes the following steps: S1.1. Clarify the type of virtual environment and the user's behavioral goals in it; S1.2. Select the input device for hand motion recognition according to the requirements; S1.3. When the user touches the virtual object, the virtual environment provides a visual response.

[0006] As a further improvement of the technical solution, in S3, the user's hand movements are processed and analyzed in real time, the user's intentions are identified and the user's hand movements are responded to, and the state of the virtual environment is adjusted according to the user's hand movements, including the following steps: S3.1. Use the deep learning YOLO model to detect the hand area and extract the region of interest; S3.2. Use the key point detection OpenPose model to extract the key point information of hand movements; S3.3, preprocessing the key point information data, and converting the key point coordinates of the hand movement into a unified reference coordinate system; S3.4, extracting static hand motion features and dynamic hand motion features, and splicing the static hand motion features and the dynamic hand motion features together to form a unified feature vector; S3.5, classify the unified feature vector using a deep learning model; S3.6. Map the classification results to the corresponding semantic intent.

[0007] As a further improvement of the technical solution, in S3.1, the deep learning YOLO model is used to detect the hand area and extract the region of interest, including the following steps: S3.11, preprocessing the user's hand motion image; S3.12, input the preprocessed image to the deep learning YOLO model, and the deep learning YOLO model outputs multiple bounding boxes; S3.13, remove redundant overlapping bounding boxes through the NMS algorithm and retain the detection box with the highest confidence; S3.14. Crop the hand region from the original user hand motion image according to the detected bounding box coordinates.

[0008] As a further improvement of the technical solution, in S3.2, the key point detection OpenPose model is used to extract the key point information of the hand movement, including the following steps: S3.21, input the obtained hand area image into the key point detection OpenPose model; S3.22, through key point detection, the OpenPose model generates the connection relationship between different joints and the probability distribution of each key point on the image; S3.23, perform peak detection on the probability distribution of each key point on the image and set a confidence threshold n; S3.24, using the connection relationship between different joints to match the connection between adjacent key points, and constructing the hand skeleton by maximizing the path score; S3.25, calculate the geometric features of joint angles and finger lengths based on the hand skeleton, and extract key point trajectories in the time series; S3.26. Analyze the hand movement type based on the key point shape and recognize the movement in combination with the key point trajectory.

[0009] As a further improvement of the technical solution, in S3.22, the connection relationship between different joints and the probability distribution of each key point on the image are: The connection relationship between different joints: ; The probability distribution of each key point on the image: ; in, Represents pixel coordinates The partial association field (PAF) value of the joint The direction vector at that position; Represents the pixel coordinates in the image; Indicates from the joint To another joint The direction vector of Indicates from the joint To another joint The connecting line segment of Indicates joints; Indicates another joint; Represented in pixel coordinates Key Points The probability value of Represents the standard deviation of the Gaussian distribution, which controls the spread of the probability distribution in the heat map; Indicates key points The real position coordinates; Indicates key points.

[0010] As a further improvement of the technical solution, in S3.25, the geometric features of the joint angle and the finger length are: Joint Angle: ; Finger length: ; in, Represents the angle between the lines of the joints; represents the vector between the first joints; represents the vector between the second joints; Indicates finger length; Indicates key points and exist Coordinate difference in the axis direction; Indicates key points and exist Coordinate difference in the axis direction; and Indicates key points.

[0011] As a further improvement of the technical solution, in S3.25, extracting key point trajectories in the time series includes the following steps: S3.251. For each key point, the position of the key point in each frame is used as time series data, and the trajectory of each key point is smoothed using a filtering algorithm. The filtering algorithm is optimized for the case where the hand movement changes rapidly. S3.252, evaluate the motion trajectory of key points in the time series by calculating the displacement, velocity and acceleration of each key point; S3.253, using key point trajectories as input features, combined with displacement, velocity and acceleration, to perform hand motion recognition; S3.254. Output the position change of each key point in the time series.

[0012] As a further improvement of the technical solution, in S3.251, the filtering algorithm is: ; in, Indicates the key point of the hand movement at this moment The smooth position of Indicates the key point of the hand movement at this moment The original location of Indicates the key point of the hand movement at the last moment The smooth position of represents the smoothing factor; Indicates a point in time; Indicates key points; In view of the situation where the hand movements change quickly, the filtering algorithm is optimized: ; in, Indicates the key points of hand movements after optimization at this moment The smooth position of It represents the adjustment coefficient; Indicates the range of change of hand movements.

[0013] As a further improvement of the technical solution, in S4, mapping the recognized hand motion with the operation in the virtual environment includes the following steps: S4.1. Map the coordinate system of the hand motion to the coordinate system of the virtual environment and design the input range of the hand motion; S4.2. Design the mapping between hand movements and virtual operations according to the application scenarios; S4.3, controlling virtual objects through recognized hand movements; S4.4. Select buttons on the virtual interface by “pointing” the hand action, and the duration of the hand action is used as the basis for triggering events; use the “grab” action in gesture recognition to grab virtual objects, and control the drag of objects by moving the hand; scroll the virtual menu by “swiping” gesture; S4.5. The recognized hand motion data is mapped to the operation in the virtual environment in real time.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. In this virtual reality human-computer interaction method, the YOLO model and the OpenPose model are combined to achieve accurate recognition and real-time response to the user's hand movements. The static features of hand movements (such as joint angles, palm normals) and dynamic features (such as motion trajectories, speed, and acceleration) are comprehensively utilized to provide an intuitive and natural interactive experience. Users can complete operations such as "grabbing" and "pointing" through intuitive gestures without the need for additional equipment, which enhances the immersion and interactivity of virtual reality.

[0015] 2. In this virtual reality human-computer interaction method, a variety of optimization technologies are introduced in the hand motion recognition process, such as denoising and smoothing key point data through filtering algorithms, optimizing bounding box detection accuracy, and combining deep learning models (ResNet) to classify gestures, thereby significantly improving recognition accuracy and robustness. In addition, through the multi-level mapping design of hand motions and virtual environment operations, it can adapt to different application scenarios (such as games, training, simulation, etc.), meet diverse needs, and improve the adaptability and practicality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example: See Figure 1 As shown, this embodiment provides a virtual reality human-computer interaction method, comprising the following steps: S1. Design an interactive interface that fits the virtual environment and ensures that users can interact with virtual objects or systems intuitively; In this embodiment, designing an interactive interface that conforms to the virtual environment includes the following steps: S1.1. Clarify the type of virtual environment (e.g., game, training, simulation, etc.) and the user's behavioral goals in it. For example, in a training scenario, the interactive interface should focus on simplicity and task guidance; while in a game scenario, it needs to focus on fun and immersion; S1.2. Select the input device for hand motion recognition according to the needs, ensure that the interaction method is consistent with the user's natural movements or habits, and simplify user operations. For example, using hand motion control allows users to point or click virtual objects with their fingers to operate; S1.3. When the user touches or approaches a virtual object, the virtual environment provides a visual response, such as a button changing color, an object highlighting, or the virtual object moving.

[0019] S2, collecting user hand movements through a camera; S3, through the key point detection OpenPose model to process and analyze the user's hand movements in real time and identify the intention of the user's hand movements; In this embodiment, the user's hand movements are processed and analyzed in real time, the user's intentions are identified and the user's hand movements are responded to, and the state of the virtual environment is adjusted according to the user's hand movements, including the following steps: S3.1. Use the deep learning YOLO model to detect the hand area and extract the region of interest; The principle of using the deep learning YOLO model to detect the hand area and extract the region of interest is to divide the input image into grids through a single-stage network structure, predict the bounding box, confidence and category probability of the target for each grid at the same time, and extract the hand area with high confidence as the region of interest (ROI). In virtual reality interaction, the hand area is the core of the interaction. The YOLO model can provide accurate input for subsequent key point detection (such as OpenPose) by efficiently locating the hand area, avoiding interference from irrelevant background. Accurately extracting the region of interest (ROI) can reduce the amount of calculation and interference, and improve the efficiency and accuracy of subsequent key point detection and gesture recognition. Use the deep learning YOLO model to detect the hand area and extract the region of interest, including the following steps: S3.11, preprocessing the user's hand motion image, including data normalization and scaling; S3.12, input the preprocessed image to the deep learning YOLO model, and the deep learning YOLO model outputs multiple bounding boxes; S3.13, remove redundant overlapping bounding boxes through the NMS algorithm, and retain the detection box with the highest confidence. The NMS algorithm calculates the intersection over union (IoU) between the detection boxes, removes low-confidence boxes with high overlap, and retains the box with the highest confidence, thereby reducing duplicate detection; S3.14. Crop the hand region from the original user hand motion image according to the detected bounding box coordinates, scale the cropped image to the input size required by the deep learning YOLO model (e.g., 128×128128\times128128×128 or 256×256256\times256256×256), and normalize the pixel values. S3.2. Use the key point detection OpenPose model to extract the key point information of hand movements, including finger joints and palm positions; Among them, the principle of using the key point detection OpenPose model to extract the key point information of hand movements is to process the input image through a convolutional neural network (CNN), predict the key point position of the hand, and infer the joint structure and movement of the hand through association; by extracting the key point information of the hand, the details of the gesture (such as the degree of finger bending, palm shape, etc.) can be accurately analyzed to provide accurate data support for subsequent gesture recognition and virtual operation; the time series data provided by OpenPose can be used to analyze the dynamic changes of hand movements, such as waving arms, grabbing objects, etc., which is helpful for action recognition and control in virtual reality interaction; accurate hand key point information enables the virtual reality system to respond to user actions more naturally and achieve a higher quality human-computer interaction experience; Using the key point detection OpenPose model to extract the key point information of hand movements includes the following steps: S3.21. Input the obtained hand region image into the key point detection OpenPose model. The OpenPose hand model outputs the 2D coordinates of 21 key points for each hand and performs regional analysis (OpenPose predicts key points for each hand separately). S3.22, through key point detection, the OpenPose model generates the connection relationship between different joints and the probability distribution of each key point on the image; Furthermore, the connection relationship between different joints and the probability distribution of each key point on the image are: The connection relationship between different joints: ; The probability distribution of each key point on the image: ; in, Represents pixel coordinates The partial association field (PAF) value of the joint The direction vector at that position; Represents the pixel coordinates in the image; Indicates from the joint To another joint The direction vector of Indicates from the joint To another joint The connecting line segment of Indicates joints; Indicates another joint; Represented in pixel coordinates Key Points The probability value of Represents the standard deviation of the Gaussian distribution, which controls the spread of the probability distribution in the heat map; Indicates key points The real position coordinates; Indicates key points; S3.23, perform peak detection on the probability distribution of each key point on the image to find the most likely position, set the confidence threshold n, and each key point position on the image has a confidence score, which reflects the certainty of the model's prediction of the key point position; S3.24, using the connection relationship between different joints to match the connection between adjacent key points, and constructing the hand skeleton by maximizing the path score; S3.25. Calculate the geometric features of joint angles and finger lengths based on the hand skeleton, and extract key point trajectories in the time series for analyzing hand movements. Joint angles are used to determine the degree of finger bending. For example, when making a fist, the joint angles of the fingers are small, while when the palm is open, the joint angles are large. Finger length is used to determine whether the fingers are fully extended. When the fingers are fully extended, the distance between the joints is long, while when the fingers are bent or retracted, the distance is short. Furthermore, the geometric characteristics of joint angles and finger lengths are: Joint Angle: ; Finger length: ; in, Represents the angle between the lines of the joints; represents the vector between the first joints; represents the vector between the second joints; Indicates finger length; Indicates key points and exist Coordinate difference in the axis direction; Indicates key points and exist Coordinate difference in the axis direction; and Indicates key points; Furthermore, extracting key point trajectories in the time series includes the following steps: The purpose of extracting key point trajectories in time series is to capture the dynamic characteristics of hand movements (such as displacement, velocity, acceleration) by analyzing the position changes of key points in consecutive frames, so as to accurately identify the motion trajectory and action type of gestures; S3.251, for each key point, the position of the key point in each frame is used as time series data, and the trajectory of each key point is smoothed using a filtering algorithm; Furthermore, filtering can make the hand trajectory smoother, reduce discontinuous jumps, facilitate subsequent action recognition and gesture analysis, and remove noise introduced by factors such as camera jitter and lighting changes, making the key point trajectory smoother and avoiding unnecessary misidentification and jitter. The smoothing algorithm helps reduce trajectory errors caused by sensor errors or environmental interference, and improves the stability and responsiveness of the overall system. The smoothed trajectory is more in line with natural movement patterns, enhancing the user's interactive experience in virtual reality and avoiding inaccurate or unsmooth operations caused by trajectory jitter. The filtering algorithm is: ; in, Indicates the key point of the hand movement at this moment The smooth position of Indicates the key point of the hand movement at this moment The original location of Indicates the key point of the hand movement at the last moment The smooth position of represents the smoothing factor; Indicates a point in time; Indicates key points; When hand movements change quickly, standard filtering algorithms may not be able to keep up in time, resulting in a lag in the system's response, or over-smoothing of fast-changing movements and loss of movement accuracy. The optimized algorithm can maintain the accuracy of fast gestures while avoiding the impact of the smoothing process on dynamic movements. In interactive scenarios such as virtual reality, user movements often require immediate feedback. Fast hand movements, such as fast waving and grabbing, require the system to respond accurately in a short time. Optimizing filtering algorithms can reduce delays and ensure a smooth and real-time interactive experience. Optimizing fast-changing hand movements can ensure that the filtering algorithm only removes noise without affecting the essence of the movement, and will not ignore important movement details due to over-smoothing, thereby ensuring the accuracy of movement recognition. Fast hand movements are often manifestations of specific gestures or actions, such as waving and clicking. Optimizing filtering algorithms can enable the system to accurately identify user intentions in complex dynamic scenes, thereby improving the interactive experience in virtual reality or augmented reality. In view of the situation where the hand movements change quickly, the filtering algorithm is optimized: ; in, Indicates the key points of hand movements after optimization at this moment The smooth position of It represents the adjustment coefficient; Indicates the amplitude of hand movements; S3.252, evaluate the motion trajectory of key points in the time series by calculating the displacement (Euclidean distance between two consecutive frames), velocity (instantaneous velocity of the key point, calculated based on displacement and time interval) and acceleration (rate of change of velocity, calculated as the difference of velocity) of each key point; S3.253, using key point trajectories as input features, combined with displacement, velocity and acceleration, to perform hand motion recognition; S3.254, output the position change of each key point in the time series; S3.26, Analyze the hand action type (such as "clenching fist", "opening palm") according to the key point shape, and identify the action (such as "waving", "grasping") in combination with the key point trajectory; S3.3. Preprocessing the key point information data includes using a filtering algorithm to denoise and smooth the key point data, and converting the key point coordinates of the hand motion to a unified reference coordinate system (such as a virtual environment coordinate system) to ensure consistency of multi-device input; S3.4, extracting static hand motion features and dynamic hand motion features, where the static hand motion features include joint angles, palm normals, finger curvature, etc., and the dynamic hand motion features include motion trajectory, speed, and acceleration, etc., and splicing the static hand motion features and the dynamic hand motion features together to form a unified feature vector; S3.5. Use a deep learning model (ResNet) to classify the unified feature vector. The principle of using a deep learning model (ResNet) to classify the unified feature vector is to learn the complex relationship between features through the skip connection and multi-layer convolution operation in the residual network (ResNet), so as to efficiently classify the input feature vector; S3.6. Map the classification results to corresponding semantic intents (such as “select an object”, “move an object”, “open a menu”).

[0020] S4, adjusting the state of the virtual environment according to the intention of the user's hand movement; In this embodiment, mapping the recognized hand motions with the operations in the virtual environment includes the following steps: S4.1. Map the coordinate system of the hand movements to the coordinate system of the virtual environment. For example, the position or movement direction of the virtual object can be controlled by the spatial position of the hand, and the input range of the hand movements can be designed to ensure that the user can interact in a reasonable space. Usually, the user's hand position is mapped to the corresponding position in the virtual world through the 3D coordinate system of the virtual environment. S4.2. Design the mapping of hand movements and virtual operations according to the application scenarios, for example, "clenching a fist": starting or stopping the operation of a virtual object (such as controlling the grabbing or releasing of a virtual object); "opening your hand": enlarging or reducing a virtual object; "waving": switching virtual scenes or turning pages; "pointing": selecting or activating virtual interface elements, such as buttons or menu items; S4.3. Control the position, rotation, and scale of virtual objects through recognized hand movements, for example: Position control: use the spatial position of the hand to control the movement of the virtual object; Rotation control: control the rotation angle of the object by rotating the fingers or wrist; Scale control: control the size of the virtual object by a "pinch" gesture (such as moving two fingers closer or farther away); S4.4. Select buttons or objects on the virtual interface by "pointing" or "clicking" hand movements, with the duration of the hand movement or the pressing action as the basis for triggering the event; use the "grab" action in gesture recognition to grab virtual objects, and control the dragging of objects by moving the hand; scroll virtual menus or adjust the settings of the virtual environment by "swiping" or "rotating" gestures; S4.5. The recognized hand motion data is mapped to the operations in the virtual environment in real time to ensure that the response time between input and output is minimized to avoid delays that affect the user experience.

[0021] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A virtual reality human-computer interaction method, characterized in that: The following steps are involved: S1. Design an interactive interface that fits the virtual environment; S2, collecting user hand movements through a camera; S3, through the key point detection OpenPose model to process and analyze the user's hand movements in real time and identify the intention of the user's hand movements; S4. Adjust the state of the virtual environment according to the intention of the user's hand movement.

2. The virtual reality human-computer interaction method according to claim 1, characterized in that: In S1, designing an interactive interface that conforms to the virtual environment includes the following steps: S1.

1. Clarify the type of virtual environment and the user's behavioral goals in it; S1.

2. Select the input device for hand motion recognition according to the requirements; S1.

3. When the user touches the virtual object, the virtual environment provides a visual response.

3. The virtual reality human-computer interaction method according to claim 2, characterized in that: In S3, the user's hand movements are processed and analyzed in real time, the user's intentions are identified and the user's hand movements are responded to, and the state of the virtual environment is adjusted according to the user's hand movements, including the following steps: S3.

1. Use the deep learning YOLO model to detect the hand area and extract the region of interest; S3.

2. Use the key point detection OpenPose model to extract the key point information of hand movements; S3.3, preprocessing the key point information data, and converting the key point coordinates of the hand movement into a unified reference coordinate system; S3.4, extracting static hand motion features and dynamic hand motion features, and splicing the static hand motion features and the dynamic hand motion features together to form a unified feature vector; S3.5, classify the unified feature vector using a deep learning model; S3.

6. Map the classification results to the corresponding semantic intent.

4. The virtual reality human-computer interaction method according to claim 3, characterized in that: In S3.1, the deep learning YOLO model is used to detect the hand area and extract the region of interest, including the following steps: S3.11, preprocessing the user's hand motion image; S3.12, input the preprocessed image to the deep learning YOLO model, and the deep learning YOLO model outputs multiple bounding boxes; S3.13, remove redundant overlapping bounding boxes through the NMS algorithm and retain the detection box with the highest confidence; S3.

14. Crop the hand region from the original user hand motion image according to the detected bounding box coordinates.

5. The virtual reality human-computer interaction method according to claim 4, characterized in that: In S3.2, the key point detection OpenPose model is used to extract the key point information of the hand movement, including the following steps: S3.21, input the obtained hand area image into the key point detection OpenPose model; S3.22, through key point detection, the OpenPose model generates the connection relationship between different joints and the probability distribution of each key point on the image; S3.23, perform peak detection on the probability distribution of each key point on the image and set a confidence threshold n; S3.24, using the connection relationship between different joints to match the connection between adjacent key points, and constructing the hand skeleton by maximizing the path score; S3.25, calculate the geometric features of joint angles and finger lengths based on the hand skeleton, and extract key point trajectories in the time series; S3.

26. Analyze the hand movement type based on the key point shape and recognize the movement in combination with the key point trajectory.

6. The virtual reality human-computer interaction method according to claim 5, characterized in that: In S3.22, the connection relationship between different joints and the probability distribution of each key point on the image are: The connection relationship between different joints: ； The probability distribution of each key point on the image: ； in, Represents pixel coordinates The partial association field (PAF) value of the joint The direction vector at that position; Represents the pixel coordinates in the image; Indicates from the joint To another joint The direction vector of Indicates from the joint To another joint The connecting line segment of Indicates joints; Indicates another joint; Represented in pixel coordinates Key Points The probability value of Represents the standard deviation of the Gaussian distribution, which controls the spread of the probability distribution in the heat map; Indicates key points The real position coordinates; Indicates key points.

7. The virtual reality human-computer interaction method according to claim 6, characterized in that: In S3.25, the geometric features of joint angles and finger lengths are: Joint Angle: ； Finger length: ； in, Represents the angle between the lines of the joints; represents the vector between the first joints; represents the vector between the second joints; Indicates finger length; Indicates key points and exist Coordinate difference in the axis direction; Indicates key points and exist Coordinate difference in the axis direction; and Indicates key points.

8. The virtual reality human-computer interaction method according to claim 7, characterized in that: In S3.25, extracting key point trajectories in the time series includes the following steps: S3.

251. For each key point, the position of the key point in each frame is used as time series data, and the trajectory of each key point is smoothed using a filtering algorithm. The filtering algorithm is optimized for the case where the hand movement changes rapidly. S3.252, evaluate the motion trajectory of key points in the time series by calculating the displacement, velocity and acceleration of each key point; S3.253, using key point trajectories as input features, combined with displacement, velocity and acceleration, to perform hand motion recognition; S3.

254. Output the position change of each key point in the time series.

9. The virtual reality human-computer interaction method according to claim 8, characterized in that: In S3.251, the filtering algorithm is: ； in, Indicates the key point of the hand movement at this moment The smooth position of Indicates the key point of the hand movement at this moment The original location of Indicates the key point of the hand movement at the last moment The smooth position of represents the smoothing factor; Indicates a point in time; Indicates key points; In view of the situation where the hand movements change quickly, the filtering algorithm is optimized: ； in, Indicates the key points of hand movements after optimization at this moment The smooth position of It represents the adjustment coefficient; Indicates the range of change of hand movements.

10. The virtual reality human-computer interaction method according to claim 9, characterized in that: In S4, mapping the recognized hand motion with the operation in the virtual environment includes the following steps: S4.

1. Map the coordinate system of the hand motion to the coordinate system of the virtual environment and design the input range of the hand motion; S4.

2. Design the mapping between hand movements and virtual operations according to the application scenarios; S4.3, controlling virtual objects through recognized hand movements; S4.

4. Select buttons on the virtual interface by "pointing" hand movements, with the duration of the hand movements serving as the basis for triggering events; use the "grab" movement in gesture recognition to grab virtual objects, and control the dragging of objects by moving the hand; scroll the virtual menu by "swiping" gestures; S4.

5. The recognized hand motion data is mapped to the operation in the virtual environment in real time.

Images