Webxr gesture interaction method and device based on state machine

Abstract

The application discloses a state machine-based WebXR gesture interaction method and device, which comprises the following steps: acquiring hand data of a user in real time through a WebXR device, performing gesture recognition on the hand data, and obtaining a gesture recognition result; inputting the gesture recognition result into a state machine, determining a target interaction state based on a current interaction state and the gesture recognition result through the state machine, and switching to the target interaction state; under the target interaction state, performing operation data calculation on the hand data according to a gesture mapping algorithm corresponding to the target interaction state, determining target 3D model operation data represented by the hand data; performing interactive operation on a target 3D model according to the model operation data, and driving a 3D rendering engine to update a visualized picture.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a WebXR gesture interaction method and device based on a state machine. Background Technology

[0002] With the rapid development of virtual reality (VR) and augmented reality (AR) technologies, WebXR, as a key technology for achieving immersive experiences in web environments, provides developers with cross-platform and cross-device XR application development capabilities. This allows users to directly access rich XR content through their browsers, greatly promoting the popularization and application of XR technology across various fields. In WebXR applications, achieving natural, efficient, and intuitive gesture interaction is a crucial aspect of enhancing the user experience. It allows users to break free from the limitations of traditional input devices and interact more freely with 3D models in the virtual environment.

[0003] Currently, while some research achievements and interaction methods have been developed in the field of WebXR gesture interaction, several shortcomings remain to be addressed. On one hand, existing gesture interaction methods lack effective logical management mechanisms when handling complex gestures and interaction state transitions. Different gestures may correspond to multiple operations in different interaction scenarios, and existing methods struggle to clearly and accurately define the relationship between gestures and interaction states. This leads to confusion during state transitions, making it difficult to accurately determine the subsequent target interaction state based on the current interaction state, thus affecting the accuracy and smoothness of the interaction. For example, when a user uses different gestures in a virtual scene with multiple operation options, existing methods may fail to accurately determine the user's intent, resulting in incorrect state transitions and causing stuttering or unresponsiveness in the interaction process.

[0004] On the other hand, existing technologies lack targeted and efficient algorithms when converting gesture data into specific operational data for 3D models. Users expect different effects from hand gestures under different interaction states. Existing methods often employ uniform or simplistic gesture data processing approaches, failing to accurately calculate and convert the acquired hand data based on the characteristics and needs of the current interaction state. This results in inaccurate operational data for the generated target 3D model, failing to truly reflect the user's intentions. For example, when fine rotation of the 3D model is required, existing methods may not be able to accurately calculate the model's rotation angle and direction based on subtle changes in hand movements, leading to a significant deviation between the operation result and the user's expectations, reducing the immersiveness and usability of the interaction. Summary of the Invention

[0005] In view of this, embodiments of this application provide a WebXR gesture interaction method and apparatus based on a state machine.

[0006] According to one aspect of this application, a state machine-based WebXR gesture interaction method is provided, the method comprising: The user's hand data is acquired in real time through a WebXR device, and gesture recognition is performed on the hand data to obtain gesture recognition results; The gesture recognition result is input into the state machine, which determines the target interaction state based on the current interaction state and the gesture recognition result, and then switches to the target interaction state. In the target interaction state, the hand data is used to perform operation data calculation based on the gesture mapping algorithm corresponding to the target interaction state to determine the target 3D model operation data represented by the hand data; The target 3D model is interactively manipulated based on the model operation data, and the 3D rendering engine is driven to update the visualization.

[0007] According to another aspect of this application, a state machine-based WebXR gesture interaction device is provided, the device comprising: The gesture recognition module is used to acquire the user's hand data in real time through the WebXR device, and to perform gesture recognition on the hand data to obtain the gesture recognition result; The state machine management module is used to input the gesture recognition result into the state machine, determine the target interaction state based on the current interaction state and the gesture recognition result through the state machine, and switch to the target interaction state. The mapping execution module is used to perform operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state in the target interaction state, determine the target 3D model operation data represented by the hand data, perform interactive operation execution on the target 3D model according to the model operation data, and drive the 3D rendering engine to update the visualization screen.

[0008] According to another aspect of this application, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described state machine-based WebXR gesture interaction method.

[0009] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described state machine-based WebXR gesture interaction method.

[0010] By employing the above technical solutions, this application provides a WebXR gesture interaction method and apparatus based on a state machine. First, by managing the interaction state through a state machine, it can accurately switch between different states based on user gestures, avoiding state confusion and improving the accuracy and smoothness of the interaction, allowing users to interact with the virtual environment more clearly and systematically. Second, by using corresponding gesture mapping algorithms to calculate operation data for different interaction states, it can accurately convert user gestures into specific operations on the 3D model, ensuring that the operation results meet user expectations and improving the accuracy and practicality of the interaction. Finally, by executing the interaction operation in a timely manner based on the operation data and driving the 3D rendering engine to update the screen, it achieves real-time feedback of the operation, allowing users to immediately see the effect of the gesture operation, greatly enhancing the immersion of the interaction and the user experience, and providing a more natural and efficient gesture interaction solution for WebXR applications.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The diagram illustrates a flowchart of a state machine-based WebXR gesture interaction method provided in an embodiment of this application. Figure 2 A flowchart illustrating another WebXR gesture interaction method based on a state machine provided in an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of the structure of a WebXR gesture interaction device based on a state machine, according to an embodiment of this application. Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0013] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0014] This embodiment provides a WebXR gesture interaction method based on a state machine, such as... Figure 1 As shown, the method includes: Step 101: Acquire the user's hand data in real time through a WebXR device, and perform gesture recognition on the hand data to obtain the gesture recognition result.

[0015] In this embodiment, the WebXR device has the ability to capture user hand movements, collecting data such as the hand's position, posture, and movement trajectory in three-dimensional space in real time. This data forms the basis for subsequent gesture interaction. Through a specific gesture recognition algorithm, this raw hand data is analyzed and processed to identify the specific gesture the user is currently making, such as pinching or chopping, thus obtaining the gesture recognition result. Specifically, gesture recognition can be performed using a pre-trained gesture recognition model.

[0016] Step 102: Input the gesture recognition result into the state machine, and use the state machine to determine the target interaction state based on the current interaction state and the gesture recognition result, and switch to the target interaction state.

[0017] In this embodiment, a state machine is a model used to manage different states of the system, responsible for managing various states during the WebXR interaction process. The current interaction state represents the current operational stage of the system. After the gesture recognition result is input into the state machine, the state machine analyzes the operational intent represented by the gesture in the current state based on preset rules and logic, combined with the current interaction state, and then determines the target interaction state that the system should switch to. This step ensures that the system can accurately and orderly transition between different interaction states based on the user's gestures, making the interaction process clearer and more organized.

[0018] Optionally, in this embodiment of the application, the state machine predefines multiple interaction states, each corresponding to a different interaction state triggering gesture type. These multiple interaction states include at least two of the following: default viewing state, object selection state, panning operation state, rotation operation state, zooming operation state, and data slicing state. The step of inputting the gesture recognition result into the state machine, determining the target interaction state based on the current interaction state and the gesture recognition result, and switching to the target interaction state includes: The gesture recognition type in the gesture recognition result is input into the state machine. The state machine determines the target interaction state trigger gesture type corresponding to the current interaction state based on the interaction state trigger gesture type corresponding to each of the multiple interaction states, and determines whether the gesture recognition type belongs to the target interaction state trigger gesture type. If it is, then trigger the gesture type according to the target interaction state, determine the target interaction state corresponding to the gesture recognition type, and switch to the target interaction state; If it does not belong to the target interaction state, then the current interaction state is determined to be the target interaction state, and the current interaction state remains unchanged.

[0019] In this embodiment, the state machine predefines various interaction states (such as default viewing state, object selection state, translation / rotation / zoom operation state, data slicing state, etc.) and configures a unique interaction state trigger gesture type for each state. The interaction state trigger gesture type for different interaction states indicates which gesture type can trigger the switching of the corresponding interaction state in the corresponding interaction state. For example, in the default viewing state, a single-hand pinch gesture triggers the translation operation state, while a two-hand pinch gesture triggers the zoom operation state. Once the system receives the gesture recognition result (including the gesture recognition type, such as "click," "drag," and "rotate"), the state machine first queries the target interaction state trigger gesture type corresponding to the current interaction state (e.g., "default viewing state"). It compares the input gesture recognition type with the target trigger gesture type for the current state. If they match, it indicates the user intends to trigger a state transition. The state machine determines the target interaction state corresponding to the gesture based on a predefined mapping relationship (e.g., a "click" gesture in the "default viewing state" might trigger an "object selection state") and executes the state switch. If they don't match, it means the current gesture has no clear intention to transition in the current state, and the state machine maintains the current state (e.g., detecting a "rotate" gesture in the "pan operation state" does not switch the state because it doesn't match the trigger type). Through the pre-configuration and dynamic matching logic of the state machine, precise control of state transitions in WebXR gesture interactions is achieved, providing a reliable state management foundation for subsequent operation data calculation and rendering updates, ultimately improving the accuracy of the interaction and the user experience.

[0020] In this embodiment of the application, optionally, the gesture recognition result includes the gesture recognition type and the confidence level corresponding to the gesture recognition type; the switching to the target interaction state includes: If the confidence level is greater than or equal to a preset first threshold, then the target interaction state is directly switched to after the target interaction state is determined. If the confidence level is less than a preset first threshold and greater than or equal to a preset second threshold, then after determining the target interaction state, an interaction state switching prompt message is output based on the target interaction state, and the system switches to the target interaction state if no feedback signal for rejecting the state switch is received; wherein, the interaction state switching prompt message is used to prompt the user to switch to the target interaction state and the feedback method for rejecting the state switch. If the confidence level is less than a preset second threshold, then interaction state switching guidance information is output based on the target interaction state, and the user switches to the target interaction state upon receiving a feedback signal that allows the state to switch; wherein, the interaction state switching guidance information is used to guide the user to make standard gestures and provide feedback on the allowed state switching method, and the feedback signal that allows the state to switch includes a signal that provides feedback in the allowed state switching method, or the gesture recognition type that identifies a confidence level greater than or equal to the preset second threshold.

[0021] In this embodiment, when the confidence level of the gesture recognition result reaches or exceeds a preset first threshold, it indicates that the system's gesture recognition is very accurate, with a high degree of certainty that the gesture is the identified type. In this case, once the target interaction state is determined, the system will directly switch to that target interaction state without requiring additional confirmation from the user, achieving a fast and smooth state transition and improving interaction efficiency. If the confidence level of the gesture recognition result is between the preset first threshold and the preset second threshold, it indicates that the system's gesture recognition has a certain degree of accuracy, but the certainty is not particularly high. At this time, after determining the target interaction state, the system will not switch immediately, but will first output an interaction state switching prompt based on the target interaction state. This prompt will inform the user of the target interaction state that the system is about to switch to, and at the same time inform the user of the feedback method for refusing the state switch. If the system does not receive a rejection signal from the user according to the prompt within a certain period of time, then the system will switch to the target interaction state. This approach considers both the accuracy of the system's recognition and gives the user a certain degree of autonomy in decision-making, avoiding unnecessary state switching due to misrecognition. When the confidence level of the gesture recognition result is lower than a preset second threshold, it indicates that the system's recognition accuracy is low and a misrecognition is highly likely. In this case, the system outputs interaction state switching guidance information based on the target interaction state. The guidance information guides the user to make standard gestures, helping the user to complete gesture operations more correctly, while informing the user of the feedback method for allowing state switching. Only when the system receives a signal from the user allowing state switching according to the guidance method, or when the system recognizes the gesture type again with a confidence level greater than or equal to the preset second threshold, will the system switch to the target interaction state. This strategy further reduces the possibility of incorrect state switching due to misrecognition, improving the accuracy and reliability of the interaction. This layered processing approach comprehensively considers the system's recognition accuracy and user needs, making gesture interaction more intelligent and user-friendly, and providing a more stable and reliable interaction solution for WebXR applications.

[0022] Step 103: In the target interaction state, perform operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state, and determine the target 3D model operation data represented by the hand data.

[0023] In this embodiment, different interaction states correspond to different operational requirements and gesture mapping relationships. After determining the target interaction state, the raw hand data is further calculated and processed according to the gesture mapping algorithm specific to that state. The gesture mapping algorithm considers the correspondence between gestures and 3D model operations in the current interaction state. For example, in the operation state, the direction and distance of finger movement may correspond to the rotation angle and direction of the 3D model. This step achieves a precise conversion from user gestures to specific 3D model operations, enabling the user's gestures to accurately control the 3D model in the virtual environment.

[0024] Step 104: Perform interactive operations on the target 3D model based on the model operation data, and drive the 3D rendering engine to update the visualization.

[0025] In this embodiment, after obtaining the target 3D model's operation data, corresponding interactive operations are performed on the target 3D model based on this data. These operations include moving the model according to a calculated translation vector, rotating the model according to a rotation angle, or adjusting the model's size according to a scaling ratio. Simultaneously, to allow users to see the operation effects in real time, the system drives the 3D rendering engine to update the visualization based on the model's new state. The 3D rendering engine is responsible for processing the geometric information, materials, lighting, and other data of the 3D model to generate a two-dimensional image that can be displayed on a display device. By updating the screen in a timely manner, users can intuitively see the results of their gesture operations, enhancing the real-time nature and immersive experience of the interaction.

[0026] By applying the technical solution of this embodiment, firstly, by managing the interaction state through a state machine, the system can accurately switch between different states based on user gestures, avoiding state confusion and improving the accuracy and smoothness of the interaction. This allows users to interact with the virtual environment more clearly and systematically. Secondly, by employing corresponding gesture mapping algorithms to calculate operation data for different interaction states, the system can accurately convert user gestures into specific operations on the 3D model, ensuring that the operation results meet user expectations and improving the precision and usability of the interaction. Finally, by executing the interaction operation in a timely manner based on the operation data and driving the 3D rendering engine to update the screen, the system achieves real-time feedback on the operation, allowing users to immediately see the effect of the gesture operation. This greatly enhances the immersiveness of the interaction and the user experience, providing a more natural and efficient gesture interaction solution for WebXR applications.

[0027] In this embodiment of the application, optionally, when the target interaction state is a translation operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state to determine the target 3D model operation data represented by the hand data, and performing interactive operation on the target 3D model according to the model operation data, includes: In the translation operation state, the following steps are performed according to the translation gesture mapping algorithm to determine the termination position of the translation operation of the target 3D model: record the collision position when the user's hand collides with the target 3D model in a single-hand pinch gesture and the initial position of the target 3D model; obtain the displacement vector of the user's hand moving in the single-hand pinch gesture; use the product of the displacement vector and the sensitivity coefficient as the displacement vector of the target 3D model; and determine the termination position of the translation operation of the target 3D model according to the displacement vector of the target 3D model and the initial position of the target 3D model. The target 3D model is moved based on the termination position of the translation operation.

[0028] In the above embodiment, when the target interaction state is determined to be a translation operation state, the hand data is processed according to a specific translation gesture mapping algorithm to complete the translation operation of the target 3D model. The specific steps are as follows: When the user makes a pinch gesture with one hand and the gesture collides with the target 3D model, the system records the collision position between the hand and the model, and also records the initial position of the target 3D model. The system continuously tracks the trajectory of the user's hand moving with a single-handed pinch gesture to obtain the displacement vector of the hand movement. This displacement vector reflects the direction and distance of the user's hand movement in space. Considering different user operating habits and device characteristics, a sensitivity coefficient is introduced. The displacement vector of the hand movement is multiplied by the sensitivity coefficient to obtain the displacement vector of the target 3D model. The sensitivity coefficient can be flexibly adjusted to adjust the magnitude of the model movement. For example, if a smaller sensitivity coefficient is desired for more precise model movement, a larger sensitivity coefficient can be set; if a faster model movement is desired, a larger sensitivity coefficient can be set. Based on the calculated displacement vector of the target 3D model and the previously recorded initial position, the final position of the target 3D model after translation is determined through simple vector operations. Based on the determined final position of the target 3D model translation, the actual movement operation is performed on the target 3D model, moving it to the corresponding position in the virtual scene. This embodiment records the collision position between the hand and the model, as well as the initial position of the model, and calculates the model displacement vector based on the hand movement displacement vector and sensitivity coefficient. This allows for precise determination of the target 3D model's movement distance and direction according to the user's actual hand movements, achieving natural and intuitive gesture interaction. Users can translate the 3D model in the virtual environment as if manipulating an object in the real world. The introduction of the sensitivity coefficient increases the flexibility and adaptability of the operation, allowing the sensitivity of model movement to be adjusted according to different scenarios and user needs, meeting diverse interaction requirements.

[0029] In this embodiment of the application, optionally, when the target interaction state is a zoom operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state to determine the target 3D model operation data represented by the hand data, and performing interactive operation on the target 3D model according to the model operation data, includes: In the zoom operation state, the following steps are performed according to the zoom gesture mapping algorithm to determine the target 3D model zoom ratio: when the user's hand is detected to be in a two-hand pinch gesture, the first distance between the first hand starting position and the second hand starting position is recorded, the second distance between the first hand ending position and the second hand ending position after the user's hand moves with the two-hand pinch gesture is obtained, and the ratio of the second distance to the first distance is calculated as the target 3D model zoom ratio. Using the midpoint of the line connecting the starting positions of the first hand and the second hand, or the midpoint of the target 3D model, as the scaling origin, the target 3D model is scaled according to the scaling ratio of the target 3D model.

[0030] In the above embodiment, when the target interaction state is in a scaling operation state, hand data is processed according to a specific scaling gesture mapping algorithm to complete the scaling operation of the target 3D model. The specific steps are as follows: The system detects the user's hand movements in real time. When the system detects that the user's hands are in a pinching gesture, it records the first hand starting position and the second hand starting position, and calculates the first distance between these two starting positions. This first distance serves as a reference value for subsequent calculation of the scaling ratio. The system continuously tracks the movement of the user's hands in the pinching gesture. When the hand movement ends, it obtains the first hand ending position and the second hand ending position, and then calculates the second distance between these two ending positions. The second distance reflects the change in distance between the user's hands after the hand movement during the scaling operation. By calculating the ratio of the second distance to the first distance, the scaling ratio of the target 3D model is obtained. This ratio directly reflects the scaling degree of the model corresponding to the user's hand movements. For example, a ratio greater than 1 indicates that the model is enlarged, and a ratio less than 1 indicates that the model is shrunk. To determine the center position for model scaling, the system can choose the midpoint of the line connecting the first and second hand starting positions as the scaling origin, or it can choose the midpoint of the target 3D model itself. Based on the calculated scaling ratio of the target 3D model, and using the determined scaling origin as the center, the system performs corresponding scaling operations on the target 3D model, enlarging or shrinking the model according to the proportion desired by the user's hand gestures. This embodiment calculates the scaling ratio by recording the initial and final distances of the user's hand pinching gestures, mapping the user's actual hand movements to the scaling operation of the target 3D model. This achieves natural and intuitive gesture interaction, allowing the user to feel as if they are directly manipulating objects in the virtual world with their hands to scale them. This improves the accuracy, smoothness, and user experience of 3D model scaling operations in WebXR applications.

[0031] Optionally, in this embodiment of the application, the step of using the midpoint of the line connecting the starting positions of the first hand and the second hand, or the midpoint of the target 3D model, as the scaling origin includes: Calculate the midpoint distance between the midpoint of the line connecting the starting positions of the first hand and the second hand and the midpoint of the target 3D model; If the midpoint distance is greater than a preset distance threshold, then the midpoint of the line connecting the first hand starting position and the second hand starting position is taken as the scaling origin. If the distance to the midpoint is less than or equal to a preset distance threshold, then the midpoint of the target 3D model is used as the scaling origin.

[0032] In this embodiment, the midpoint of the line segment connecting the starting positions of the first and second hands is calculated. Then, the midpoint of the target 3D model is obtained, and the distance between these two midpoints is calculated. The system pre-sets a preset distance threshold. The calculated midpoint distance is compared with this preset distance threshold. If the midpoint distance is greater than the preset distance threshold, it indicates that the midpoint of the line connecting the starting positions of the user's hands is relatively far from the midpoint of the target 3D model. In this case, the midpoint of the line connecting the starting positions of the first and second hands is used as the scaling origin. This setting makes the scaling operation more closely match the center desired by the user's hand movements, making the user feel that the model is being scaled with their own hands as the center. If the midpoint distance is less than or equal to the preset distance threshold, it indicates that the midpoint of the line connecting the starting positions of the user's hands is relatively close to the midpoint of the target 3D model. In this case, the midpoint of the target 3D model is used as the scaling origin. This selection method ensures that the scaling operation is based on the center of the model itself, so that the model maintains a relatively stable visual effect during scaling and avoids unnatural offsets caused by the scaling origin deviating from the model center. This embodiment calculates the distance between the midpoint of the line connecting the starting positions of both hands and the midpoint of the model, and flexibly selects the scaling origin based on a comparison of this distance with a preset threshold, fully considering user operating habits and the characteristics of the model itself. When the hands are far from the center of the model, using the midpoint of the line connecting the hands as the origin enhances the user's intuitive feeling and sense of control over the scaling operation, making the interaction more natural and smooth; when the hands are close to the center of the model, using the midpoint of the model as the origin ensures the stability and rationality of model scaling, avoiding visually abnormal offsets during model scaling.

[0033] In this embodiment of the application, optionally, when the target interaction state is a slice operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state to determine the target 3D model operation data represented by the hand data, and performing interactive operation on the target 3D model according to the model operation data, includes: In the slicing operation state, the following steps are performed according to the slicing gesture mapping algorithm to determine the cutting plane: when the user's hand is detected to be making a hand knife gesture, the center point of the palm and the normal vector of the palm are determined, and the cutting plane is determined according to the center point of the palm and the normal vector; The target 3D model is sliced ​​and trimmed according to the cutting plane in the rendering shader.

[0034] In the above embodiment, when the target interaction state is a slicing operation state, hand data is processed according to a specific slicing gesture mapping algorithm to complete the slicing operation on the target 3D model. The specific steps are as follows: The system continuously detects the user's hand movements. When the system detects that the user's hand is making a gesture similar to a hand chop, it begins to determine key elements. First, the center point of the palm is determined, which represents a core position of the hand in space; at the same time, the normal vector of the palm is determined, which reflects the orientation information of the palm. Based on the determined center point of the palm and the normal vector, the cutting plane is determined using geometric principles. The center point of the palm is a point on the cutting plane, and the normal vector determines the direction of the cutting plane. These two elements can uniquely determine a plane, which is the cutting plane used for subsequent slicing operations on the target 3D model. In the rendering shader, the determined cutting plane is used to perform slicing and trimming operations on the target 3D model. The rendering shader is a crucial part of graphics processing used for calculating and generating images. It can crop and remove portions of a target 3D model located on one side of the cutting plane based on the position and orientation of the cutting plane, thus presenting the effect of slicing the model. This embodiment determines the cutting plane by detecting the user's hand gesture and using the center point of the palm and the normal vector. This gesture interaction method is natural and intuitive, conforming to people's intuition about cutting with their hands in the real world, allowing users to easily slice the target 3D model in a virtual environment. Performing slicing and cropping within the rendering shader can efficiently and accurately achieve the slicing effect, ensuring the real-time performance and accuracy of the slicing operation.

[0035] In this embodiment of the application, optionally, when the target interaction state is a selection operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state to determine the target 3D model operation data represented by the hand data, and performing interactive operation execution on the target 3D model according to the model operation data, includes: In the selected operation state, the following steps are performed according to the selection gesture mapping algorithm to determine the selection object: when the user's hand is detected to be making an index finger pointing gesture, it is determined that a ray is emitted from the fingertip of the index finger as the origin and in the direction of the index finger pointing. The ray is then used to perform collision detection with the 3D model in the scene, and the target 3D model that collides with the ray is determined as the selection object. Set the selected object to the selected state.

[0036] In the above embodiment, when in the selection operation state, hand data is processed according to a specific selection gesture mapping algorithm to complete the selection operation of the target 3D model. The specific steps are as follows: The system continuously monitors the user's hand movements. Once it detects that the user's hand is pointing with the index finger, it emits a ray along the direction pointed by the index finger, using the fingertip as the origin. This ray represents the extension direction of the user's gaze or intention in the virtual scene. Collision detection: The emitted ray is used to perform collision detection with each 3D model in the scene. Collision detection is a commonly used technique in computer graphics to determine whether two or more objects are in contact or overlap in space. In this way, the 3D models that the ray passes through in the virtual scene can be found. Based on the collision detection results, the target 3D model that collides with the ray is determined. This model is the object that the user wants to select. The determined selection object is set to the selected state. The selected state can be reflected by changing the appearance of the model, such as highlighting or changing the color, so that the user can clearly know the currently selected model. This embodiment determines the selected object by detecting the user's index finger pointing gesture and emitting rays for collision detection. This interaction method is very natural and intuitive, which conforms to people's habit of pointing at objects with their fingers in the real world. It greatly reduces the user's learning cost and enables users to get started quickly and easily complete the selection operation in the virtual environment.

[0037] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another WebXR gesture interaction method based on a state machine is provided, such as... Figure 2 As shown, the method includes: Step 201: Based on a preset time window, calculate the variance of the user's recent hand movement speed; Step 202: If the variance is greater than or equal to the preset first variance, then set the sensitivity coefficient in the translation gesture mapping algorithm to a high sensitivity coefficient, and set the dead zone threshold of the gesture mapping algorithm to a low dead zone threshold. Step 203: If the variance is less than a preset first variance and the variance is greater than or equal to a preset second variance, then set the sensitivity coefficient in the translation gesture mapping algorithm to a medium sensitivity coefficient and set the dead zone threshold of the gesture mapping algorithm to a medium dead zone threshold. Step 204: If the variance is less than the preset second variance, then set the sensitivity coefficient in the translation gesture mapping algorithm to a low sensitivity coefficient, and set the dead zone threshold of the gesture mapping algorithm to a low dead zone threshold.

[0038] In this embodiment, the sensitivity coefficient and dead zone threshold in the translation gesture mapping algorithm can be dynamically adjusted based on the variance of the user's hand movement speed to optimize the user's gesture interaction experience in a virtual reality (WebXR) environment. Specifically, within a preset time window, the user's recent hand movement speed is monitored and recorded, and the variance of this hand movement speed data is calculated. Variance is a statistical measure of the dispersion of a set of data. By calculating the variance of hand movement speed, the fluctuation of the user's hand movement speed can be understood. The larger the variance, the greater the fluctuation of the hand movement speed, that is, the more intense, rapid, and unstable the hand movements; the smaller the variance, the more stable and gentle the hand movement speed. When the calculated variance of hand movement speed is greater than or equal to a preset first variance, it indicates that the user's hand movement speed fluctuates greatly and the movements are relatively intense. At this time, the sensitivity coefficient in the translation gesture mapping algorithm is set to a high sensitivity coefficient, which means that even small movements of the user's hand can be quickly and significantly captured by the algorithm and converted into corresponding translation operations, enabling the user to perform translation operations on objects in the virtual scene more quickly. Simultaneously, the dead zone threshold of the gesture mapping algorithm is set to a low dead zone threshold. The dead zone threshold refers to the range within which the system does not recognize hand movements as valid actions during gesture interaction, thus avoiding misoperations caused by unconscious, minor hand tremors. A low dead zone threshold makes the system more sensitive to user hand movements, accurately recognizing even very small valid movements, further enhancing the response speed and accuracy of gesture interaction. If the variance of hand movement speed is less than a preset first variance and greater than or equal to a preset second variance, it indicates that the fluctuation of the user's hand movement speed is at a moderate level. In this case, the sensitivity coefficient in the translation gesture mapping algorithm is set to a medium sensitivity coefficient, ensuring that the system's response speed and amplitude to hand movements are moderate—neither too sensitive leading to misoperations nor too sluggish affecting the user experience. At the same time, setting the dead zone threshold of the gesture mapping algorithm to a medium dead zone threshold strikes a balance between avoiding misoperations and accurately recognizing valid actions, ensuring the stability and accuracy of gesture interaction. When the variance of hand movement speed is less than a preset second variance, it indicates that the user's hand movement speed is relatively stable and smooth. At this point, the sensitivity coefficient in the translation gesture mapping algorithm is set to a low sensitivity coefficient to reduce the system's responsiveness to minute hand movements. This avoids unnecessary translation operations caused by unconscious slight hand tremors, making the operation more stable. Simultaneously, the dead zone threshold of the gesture mapping algorithm is set to a low dead zone threshold. Although hand movements are relatively stable at this point, the low dead zone threshold still ensures that the system can promptly and accurately recognize and respond when the user has a clear intention, guaranteeing the reliability of gesture interaction.

[0039] By applying the technical solution of this embodiment, the variance of the user's recent hand movement speed is statistically analyzed based on a preset time window. The sensitivity coefficient and dead zone threshold of the translation gesture mapping algorithm are dynamically adjusted according to different ranges of variance. This dynamic adjustment mechanism fully considers different states of the user's hand movements. When hand movements are vigorous, the sensitivity coefficient is increased and the dead zone threshold is decreased, enabling a quick and accurate response to user operations and enhancing the real-time performance and precision of the interaction. When hand movements are stable, the sensitivity coefficient is decreased and the dead zone threshold is set appropriately, effectively preventing misoperations and improving operational stability. This intelligent adjustment method provides users with a more natural, smooth, accurate, and stable gesture interaction experience.

[0040] Furthermore, as Figure 1 In terms of specific implementation, this application provides a WebXR gesture interaction device based on a state machine, such as... Figure 3 As shown, the device includes: The gesture recognition module is used to acquire the user's hand data in real time through the WebXR device, and to perform gesture recognition on the hand data to obtain the gesture recognition result; The state machine management module is used to input the gesture recognition result into the state machine, determine the target interaction state based on the current interaction state and the gesture recognition result through the state machine, and switch to the target interaction state. The mapping execution module is used to perform operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state in the target interaction state, determine the target 3D model operation data represented by the hand data, perform interactive operation execution on the target 3D model according to the model operation data, and drive the 3D rendering engine to update the visualization screen.

[0041] Optionally, the state machine predefines multiple interaction states, each corresponding to a specific interaction state trigger gesture type. These multiple interaction states include at least two of the following: default viewing state, object selection state, panning operation state, rotation operation state, zooming operation state, and data slicing state. The state machine management module is specifically used for: The gesture recognition type in the gesture recognition result is input into the state machine. The state machine determines the target interaction state trigger gesture type corresponding to the current interaction state based on the interaction state trigger gesture type corresponding to each of the multiple interaction states, and determines whether the gesture recognition type belongs to the target interaction state trigger gesture type. If it is, then trigger the gesture type according to the target interaction state, determine the target interaction state corresponding to the gesture recognition type, and switch to the target interaction state; If it does not belong to the target interaction state, then the current interaction state is determined to be the target interaction state, and the current interaction state remains unchanged.

[0042] Optionally, the gesture recognition result includes the gesture recognition type and the confidence level corresponding to the gesture recognition type; the state machine management module is further configured to: If the confidence level is greater than or equal to a preset first threshold, then the target interaction state is directly switched to after the target interaction state is determined. If the confidence level is less than a preset first threshold and greater than or equal to a preset second threshold, then after determining the target interaction state, an interaction state switching prompt message is output based on the target interaction state, and the system switches to the target interaction state if no feedback signal for rejecting the state switch is received; wherein, the interaction state switching prompt message is used to prompt the user to switch to the target interaction state and the feedback method for rejecting the state switch. If the confidence level is less than a preset second threshold, then interaction state switching guidance information is output based on the target interaction state, and the user switches to the target interaction state upon receiving a feedback signal that allows the state to switch; wherein, the interaction state switching guidance information is used to guide the user to make standard gestures and provide feedback on the allowed state switching method, and the feedback signal that allows the state to switch includes a signal that provides feedback in the allowed state switching method, or the gesture recognition type that identifies a confidence level greater than or equal to the preset second threshold.

[0043] Optionally, when the target interaction state is a translation operation state, the mapping execution module is specifically used to: in the translation operation state, perform the following steps according to the translation gesture mapping algorithm to determine the termination position of the translation operation of the target 3D model: record the collision position when the user's hand collides with the target 3D model with a single-hand pinch gesture and the initial position of the target 3D model, obtain the displacement vector of the user's hand moving with the single-hand pinch gesture, take the product of the displacement vector and the sensitivity coefficient as the displacement vector of the target 3D model, and determine the termination position of the translation operation of the target 3D model according to the displacement vector of the target 3D model and the initial position of the target 3D model; The target 3D model is moved based on the termination position of the translation operation.

[0044] Optionally, when the target interaction state is a scaling operation state, the mapping execution module is specifically used to: in the scaling operation state, perform the following steps according to the scaling gesture mapping algorithm to determine the scaling ratio of the target 3D model: when the user's hand is detected to be in a pinching gesture, record the first distance between the first hand starting position and the second hand starting position, obtain the second distance between the first hand ending position and the second hand ending position after the user's hand moves with the pinching gesture, and calculate the ratio of the second distance to the first distance as the scaling ratio of the target 3D model; Using the midpoint of the line connecting the starting positions of the first hand and the second hand, or the midpoint of the target 3D model, as the scaling origin, the target 3D model is scaled according to the scaling ratio of the target 3D model.

[0045] Optionally, the mapping execution module is specifically used for: Calculate the midpoint distance between the midpoint of the line connecting the starting positions of the first hand and the second hand and the midpoint of the target 3D model; If the midpoint distance is greater than a preset distance threshold, then the midpoint of the line connecting the first hand starting position and the second hand starting position is taken as the scaling origin. If the distance to the midpoint is less than or equal to a preset distance threshold, then the midpoint of the target 3D model is used as the scaling origin.

[0046] Optionally, when the target interaction state is a slicing operation state, the mapping execution module is specifically used to: in the slicing operation state, perform the following steps according to the slicing gesture mapping algorithm to determine the cutting plane: when the user's hand is detected to be making a hand-knife gesture, determine the center point of the palm and the normal vector of the palm, and determine the cutting plane according to the center point of the palm and the normal vector; The target 3D model is sliced ​​and trimmed according to the cutting plane in the rendering shader.

[0047] Optionally, when the target interaction state is a selection operation state, the mapping execution module is specifically used to: in the selection operation state, perform the following steps according to the selection gesture mapping algorithm to determine the selection object: when the user's hand is detected to be making an index finger pointing gesture, determine that a ray is emitted from the fingertip of the index finger as the origin and in the direction of the index finger pointing, perform collision detection between the ray and the 3D model in the scene, and determine the target 3D model that collides with the ray as the selection object; Set the selected object to the selected state.

[0048] Optionally, the device further includes: an adaptive adjustment module, used for: Based on a preset time window, the variance of the user's recent hand movement speed is calculated. If the variance is greater than or equal to the preset first variance, then the sensitivity coefficient in the translation gesture mapping algorithm is set to a high sensitivity coefficient, and the dead zone threshold of the gesture mapping algorithm is set to a low dead zone threshold. If the variance is less than a preset first variance and the variance is greater than or equal to a preset second variance, then the sensitivity coefficient in the translation gesture mapping algorithm is set to a medium sensitivity coefficient, and the dead zone threshold of the gesture mapping algorithm is set to a medium dead zone threshold. If the variance is less than the preset second variance, then the sensitivity coefficient in the translation gesture mapping algorithm is set to a low sensitivity coefficient, and the dead zone threshold of the gesture mapping algorithm is set to a low dead zone threshold.

[0049] It should be noted that other corresponding descriptions of the functional units involved in the state machine-based WebXR gesture interaction device provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0050] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 4 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0051] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0052] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0053] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0054] It should be noted that the user personal information involved in the embodiments of this application is all authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals. It should be noted that if any software tools or components other than those of this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.

[0055] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A WebXR gesture interaction method based on a state machine, characterized in that, The method includes: The user's hand data is acquired in real time through a WebXR device, and gesture recognition is performed on the hand data to obtain gesture recognition results; The gesture recognition result is input into the state machine, which determines the target interaction state based on the current interaction state and the gesture recognition result, and then switches to the target interaction state. In the target interaction state, the hand data is used to perform operation data calculation based on the gesture mapping algorithm corresponding to the target interaction state to determine the target 3D model operation data represented by the hand data; The target 3D model is interactively manipulated based on the model operation data, and the 3D rendering engine is driven to update the visualization.

2. The method according to claim 1, characterized in that, The state machine predefines multiple interaction states, each corresponding to a specific gesture type. These multiple interaction states include at least two of the following: default viewing state, object selection state, panning state, rotation state, zooming state, and data slicing state. The step of inputting the gesture recognition result into the state machine, determining the target interaction state based on the current interaction state and the gesture recognition result, and switching to the target interaction state includes: The gesture recognition type in the gesture recognition result is input into the state machine. The state machine determines the target interaction state trigger gesture type corresponding to the current interaction state based on the interaction state trigger gesture type corresponding to each of the multiple interaction states, and determines whether the gesture recognition type belongs to the target interaction state trigger gesture type. If it is, then trigger the gesture type according to the target interaction state, determine the target interaction state corresponding to the gesture recognition type, and switch to the target interaction state; If it does not belong to the target interaction state, then the current interaction state is determined to be the target interaction state, and the current interaction state remains unchanged.

3. The method according to claim 2, characterized in that, The gesture recognition result includes the gesture recognition type and the confidence level corresponding to the gesture recognition type; The switching to the target interaction state includes: If the confidence level is greater than or equal to a preset first threshold, then the target interaction state is directly switched to after the target interaction state is determined. If the confidence level is less than a preset first threshold and greater than or equal to a preset second threshold, then after determining the target interaction state, an interaction state switching prompt message is output based on the target interaction state, and the system switches to the target interaction state if no feedback signal for rejecting the state switch is received; wherein, the interaction state switching prompt message is used to prompt the user to switch to the target interaction state and the feedback method for rejecting the state switch. If the confidence level is less than a preset second threshold, then interaction state switching guidance information is output based on the target interaction state, and the user switches to the target interaction state upon receiving a feedback signal that allows the state to switch; wherein, the interaction state switching guidance information is used to guide the user to make standard gestures and provide feedback on the allowed state switching method, and the feedback signal that allows the state to switch includes a signal that provides feedback in the allowed state switching method, or the gesture recognition type that identifies a confidence level greater than or equal to the preset second threshold.

4. The method according to claim 2, characterized in that, When the target interaction state is a translation operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state, determining the target 3D model operation data represented by the hand data, and performing interactive operations on the target 3D model according to the model operation data includes: In the translation operation state, the following steps are performed according to the translation gesture mapping algorithm to determine the termination position of the translation operation of the target 3D model: record the collision position when the user's hand collides with the target 3D model in a single-hand pinch gesture and the initial position of the target 3D model; obtain the displacement vector of the user's hand moving in the single-hand pinch gesture; use the product of the displacement vector and the sensitivity coefficient as the displacement vector of the target 3D model; and determine the termination position of the translation operation of the target 3D model according to the displacement vector of the target 3D model and the initial position of the target 3D model. The target 3D model is moved based on the termination position of the translation operation.

5. The method according to claim 2, characterized in that, When the target interaction state is a zoom operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state, determining the target 3D model operation data represented by the hand data, and performing interactive operations on the target 3D model according to the model operation data includes: In the zoom operation state, the following steps are performed according to the zoom gesture mapping algorithm to determine the target 3D model zoom ratio: when the user's hand is detected to be in a two-hand pinch gesture, the first distance between the first hand starting position and the second hand starting position is recorded, the second distance between the first hand ending position and the second hand ending position after the user's hand moves with the two-hand pinch gesture is obtained, and the ratio of the second distance to the first distance is calculated as the target 3D model zoom ratio. Using the midpoint of the line connecting the starting positions of the first hand and the second hand, or the midpoint of the target 3D model, as the scaling origin, the target 3D model is scaled according to the scaling ratio of the target 3D model.

6. The method according to claim 5, characterized in that, The scaling origin is defined as the midpoint of the line connecting the starting positions of the first hand and the second hand, or the midpoint of the target 3D model. Calculate the midpoint distance between the midpoint of the line connecting the starting positions of the first hand and the second hand and the midpoint of the target 3D model; If the midpoint distance is greater than a preset distance threshold, then the midpoint of the line connecting the first hand starting position and the second hand starting position is taken as the scaling origin. If the distance to the midpoint is less than or equal to a preset distance threshold, then the midpoint of the target 3D model is used as the scaling origin.

7. The method according to claim 2, characterized in that, When the target interaction state is a slice operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state, determining the target 3D model operation data represented by the hand data, and performing interactive operations on the target 3D model according to the model operation data includes: In the slicing operation state, the following steps are performed according to the slicing gesture mapping algorithm to determine the cutting plane: when the user's hand is detected to be making a hand knife gesture, the center point of the palm and the normal vector of the palm are determined, and the cutting plane is determined according to the center point of the palm and the normal vector; The target 3D model is sliced ​​and trimmed according to the cutting plane in the rendering shader.

8. The method according to claim 2, characterized in that, When the target interaction state is a selection operation state, the step of performing operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state, determining the target 3D model operation data represented by the hand data, and performing interactive operation on the target 3D model according to the model operation data includes: In the selected operation state, the following steps are performed according to the selection gesture mapping algorithm to determine the selection object: when the user's hand is detected to be making an index finger pointing gesture, it is determined that a ray is emitted from the fingertip of the index finger as the origin and in the direction of the index finger pointing. The ray is then used to perform collision detection with the 3D model in the scene, and the target 3D model that collides with the ray is determined as the selection object. Set the selected object to the selected state.

9. The method according to any one of claims 4 to 8, characterized in that, The method further includes: Based on a preset time window, the variance of the user's recent hand movement speed is calculated. If the variance is greater than or equal to the preset first variance, then the sensitivity coefficient in the translation gesture mapping algorithm is set to a high sensitivity coefficient, and the dead zone threshold of the gesture mapping algorithm is set to a low dead zone threshold. If the variance is less than a preset first variance and the variance is greater than or equal to a preset second variance, then the sensitivity coefficient in the translation gesture mapping algorithm is set to a medium sensitivity coefficient, and the dead zone threshold of the gesture mapping algorithm is set to a medium dead zone threshold. If the variance is less than the preset second variance, then the sensitivity coefficient in the translation gesture mapping algorithm is set to a low sensitivity coefficient, and the dead zone threshold of the gesture mapping algorithm is set to a low dead zone threshold.

10. A WebXR gesture interaction device based on a state machine, characterized in that, The device includes: The gesture recognition module is used to acquire the user's hand data in real time through the WebXR device, and to perform gesture recognition on the hand data to obtain the gesture recognition result; The state machine management module is used to input the gesture recognition result into the state machine, determine the target interaction state based on the current interaction state and the gesture recognition result through the state machine, and switch to the target interaction state. The mapping execution module is used to perform operation data calculation on the hand data according to the gesture mapping algorithm corresponding to the target interaction state in the target interaction state, determine the target 3D model operation data represented by the hand data, perform interactive operation execution on the target 3D model according to the model operation data, and drive the 3D rendering engine to update the visualization screen.

Images