Interaction method and apparatus, storage medium, device, and program product

Abstract

The present disclosure provides an interaction method and apparatus, a storage medium, a device, and a program product. The interaction method comprises: determining that a hand switches from a first hand gesture to a second hand gesture; and in response to the second hand gesture being released at a target pose, triggering a target function corresponding to the target pose. According to the present disclosure, a hand switch of a user from a specific hand gesture (the first hand gesture) to another hand gesture (the second hand gesture) is recognized and the second hand gesture is released at a specific target pose, thereby triggering the function associated with the target pose. A natural interaction mode for triggering different functions is implemented by users switching different hand gestures and poses with one hand, thereby simplifying the interaction process; and there is no need for users to memorize complex commands or operation sequences, and function triggering can be implemented simply by changing hand gestures, thereby reducing the memory burden of the users, and improving the interaction efficiency.

Description

Interaction methods, devices, storage media, equipment and program products

[0001] This application claims priority to Chinese Patent Application No. 202410925468.5, filed on July 10, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to an interaction method, apparatus, storage medium, device, and program product. Background Technology

[0003] With the rapid development of XR technology, optimizing human-computer interaction methods has become a key factor in improving user experience. Traditional interaction methods typically involve recognizing specific, fixed gestures, followed by a menu where the user selects a button to activate a specific function. While this method provides a degree of immersion, its limitations are also quite apparent, including but not limited to the complexity of the interaction process, the difficulty for users to memorize gestures, and inefficiency. Summary of the Invention

[0004] This disclosure provides an interaction method, apparatus, storage medium, device, and program product that allows users to switch between different hand shapes and postures with one hand to trigger different functions, simplifying the interaction process, reducing the user's memory burden, and improving interaction efficiency.

[0005] On one hand, embodiments of this disclosure provide an interaction method, the method comprising: determining that a hand is switched from a first hand shape to a second hand shape; and triggering a target function corresponding to the target pose in response to the release of the second hand shape in a target pose.

[0006] On the other hand, embodiments of this disclosure provide an interactive device, the device comprising:

[0007] A determining unit is used to determine when the hand switches from a first hand shape to a second hand shape;

[0008] An interaction unit is used to trigger the target function corresponding to the target pose in response to the release of the second hand shape in the target pose.

[0009] On the other hand, embodiments of this disclosure provide a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the interactive method as described in any of the above embodiments.

[0010] On the other hand, embodiments of this disclosure provide a terminal device, the terminal device including a processor and a memory, the memory storing a computer program, and the processor executing the interaction method as described in any of the above embodiments by calling the computer program stored in the memory.

[0011] On the other hand, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the interaction method as described in any of the above embodiments. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 is a flowchart illustrating the interaction method provided in an embodiment of this disclosure;

[0014] Figure 2 is a schematic diagram of a first application scenario of the interaction method provided in the embodiments of this disclosure;

[0015] Figure 3 is a schematic diagram of a second application scenario of the interaction method provided in the embodiments of this disclosure;

[0016] Figure 4 is a schematic diagram of a third application scenario of the interaction method provided in the embodiments of this disclosure;

[0017] Figure 5 is a schematic diagram of the fourth application scenario of the interaction method provided in the embodiments of this disclosure;

[0018] Figure 6 is a schematic diagram of the fifth application scenario of the interaction method provided in the embodiments of this disclosure;

[0019] Figure 7 is a schematic diagram of the sixth application scenario of the interaction method provided in the embodiments of this disclosure;

[0020] Figure 8 is a schematic diagram of the seventh application scenario of the interaction method provided in the embodiments of this disclosure;

[0021] Figure 9 is a schematic diagram of the eighth application scenario of the interaction method provided in the embodiments of this disclosure;

[0022] Figure 10 is a schematic diagram of the ninth application scenario of the interaction method provided in the embodiments of this disclosure;

[0023] Figure 11 is a schematic diagram of the tenth application scenario of the interaction method provided in the embodiments of this disclosure;

[0024] Figure 12 is a schematic diagram of the structure of the interactive device provided in an embodiment of this disclosure;

[0025] Figure 13 is a first structural schematic diagram of a terminal device provided in an embodiment of this disclosure; and

[0026] Figure 14 is a schematic diagram of the second structure of the terminal device provided in an embodiment of this disclosure. Detailed Implementation

[0027] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] This disclosure provides an interaction method, apparatus, computer-readable storage medium, terminal device, server, and computer program product. Specifically, the interaction method of this disclosure can be executed by a terminal device or by a server.

[0029] The embodiments disclosed herein can be applied to various application scenarios such as Extended Reality (XR), Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0030] First, some of the nouns or terms that appear in the description of the embodiments of this disclosure are explained as follows:

[0031] A virtual scene is a virtual environment displayed (or provided) by an application while it is running on a terminal or server. Optionally, this virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. A virtual scene can be either a two-dimensional or three-dimensional virtual scene, and the virtual environment can be sky, land, ocean, etc., where the land includes environmental elements such as deserts and cities. The virtual scene is a scenario containing the complete game logic of virtual objects controlled by the user.

[0032] A virtual object refers to a controllable dynamic object in a virtual scene. Optionally, the dynamic object can be a virtual character, virtual animal, anime character, etc. The virtual object is a character controlled by the player through an input device, or an artificial intelligence (AI) trained and set up for battle in a virtual environment, or a non-player character (NPC) set up for battle in a virtual scene. Optionally, the virtual object is a virtual character competing in a virtual scene. Optionally, the number of virtual objects in the virtual scene battle is preset or dynamically determined according to the number of clients joining the battle; this disclosure does not limit this. In one possible implementation, the user can control the virtual object to move within the virtual scene, for example, controlling the virtual object to run, jump, crawl, etc., and can also control the virtual object to use skills, virtual items, etc., provided by the application to fight against other virtual objects. Optionally, a virtual object can also refer to an interactive static object in a virtual scene, such as a virtual object, virtual control, interface element, virtual item, etc.

[0033] Extended Reality (XR) is a concept that includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). It refers to the technology that creates an environment that connects the virtual world with the real world, allowing users to interact with that environment in real time.

[0034] Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, allowing users to immerse themselves in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0035] Augmented Reality (AR) is a technology that calculates the camera's pose parameters in the real world (or 3D world, the real world) in real time during image capture, and adds virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0036] Mixed Reality (MR) is a simulated scene that integrates computer-created sensory input (e.g., virtual objects) with sensory input or its representation from a physical setting. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0037] Augmented Virtuality (AV): An AV scene refers to a computer-created scene or virtual scene that incorporates at least one sensory input from a physical scene. The one or more sensory inputs from the physical scene can be a representation of at least one feature of the physical scene. For example, virtual objects can present the colors of physical elements captured by one or more imaging sensors. As another example, virtual objects can present features consistent with actual weather conditions in a physical scene, such as those identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest can have virtual trees and structures, but animals can have features accurately reproduced from images taken of physical animals.

[0038] Virtual field of view (FOV) is the area in a virtual environment that a user can perceive through lenses in a virtual reality device. The perceived area is represented by the field of view angle (FOV).

[0039] Augmented reality devices, which are terminals that enable augmented reality effects, can typically be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to achieve visual perception and other forms of perception. Of course, the forms that augmented reality devices can take are not limited to these, and they can be further miniaturized or enlarged as needed.

[0040] The extended reality devices described in this disclosure can include, but are not limited to, the following types:

[0041] PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions, while external PC-based extended reality devices use the data output from the PC to achieve the virtual reality effect.

[0042] Mobile extended reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connections with the mobile terminal, the mobile terminal performs calculations related to virtual reality functions and outputs data to the mobile extended reality device, such as watching virtual reality videos through an app on the mobile terminal.

[0043] All-in-one extended reality devices have processors for performing virtual functions, thus enabling independent virtual reality input and output. They do not require connection to a PC or mobile terminal, offering a high degree of freedom of use.

[0044] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0045] The embodiments of this disclosure provide an interaction method, which can be executed by a terminal or a server, or by both a terminal and a server. The embodiments of this disclosure illustrate the interaction method executed by a terminal (terminal device) as an example.

[0046] Please refer to Figures 1 to 11. Figure 1 is a flowchart illustrating the interaction method provided in this embodiment of the present disclosure, and Figures 2 to 11 are application scenario diagrams illustrating the interaction method provided in this embodiment of the present disclosure. This method can be applied to a terminal device, which may include any one of extended reality devices, virtual reality devices, augmented reality devices, and mixed reality devices. The method includes:

[0047] Step 110: Determine whether the hand position changes from the first hand shape to the second hand shape.

[0048] The first hand gesture can be different from the second hand gesture. For example, the second hand gesture can be a clenched fist. In this case, the first hand gesture is not a clenched fist. The first hand gesture can be, but is not limited to, a releasing hand gesture, a thumbs-up hand gesture, a pinching hand gesture, or any other hand gesture.

[0049] The switch from the first hand pattern to the second hand pattern can be a direct switch from the first hand pattern to the second hand pattern, or the switch can be made when the first hand pattern meets certain preset conditions (such as a short time interval).

[0050] Prior to step 110, the three-dimensional environment generated by the extended reality device may also be displayed.

[0051] For example, extended reality technology can be used to construct a realistic three-dimensional environment. This environment provides users with an immersive experience, making them feel as if they are in a real yet virtual world. The three-dimensional environment generated by extended reality devices can include both real-world and virtual-world environments. The three-dimensional environment can be a virtual reality environment or an extended reality environment.

[0052] Extended reality devices can possess powerful graphics processing capabilities to generate high-quality 3D environments in real time. This typically involves using advanced graphics rendering techniques such as ray tracing, shadow mapping, and texture mapping to create realistic 3D scenes. To provide an immersive experience, extended reality devices can also have depth perception and stereoscopic imaging capabilities. This can be achieved using technologies such as infrared sensors, depth cameras, or LiDAR, which can capture detailed information about the surrounding environment and convert it into 3D models.

[0053] When generating a 3D environment, the user's head and eye movements can also be taken into account. Augmented reality devices can have high-precision motion tracking capabilities, enabling them to capture the user's head and hand shapes, eye movements, and gestures in real time, and adjust the perspective and details of the 3D environment based on these inputs to provide a natural and immersive experience.

[0054] To enhance the user experience, the 3D environment can be integrated with the user interface. The user interface can be customized to meet the application's needs, providing various functions and controls such as menus, buttons, and input fields. These interface elements can be realistically presented within the 3D environment, allowing users to interact intuitively.

[0055] Before step 110, the pose information of the target object's head and hands can also be detected.

[0056] For example, the target audience could be users currently using extended reality devices to experience immersive experiences.

[0057] For example, sensors built into virtual reality devices (such as accelerometers, gyroscopes, magnetometers, and cameras) or external sensors (such as optical tracking systems and depth cameras) can be used to capture and calculate the user's head movements and hand movements (e.g., through handheld controllers or gesture recognition technology) in real time. This allows for the collection of head and hand position and hand shape data to obtain relevant pose information. This pose information can include head orientation, head coordinates, hand coordinates, wrist rotation angle, and hand movement speed.

[0058] In some embodiments, before determining that the hand has switched from a first hand shape to a second hand shape, the method further includes: detecting that the wrist node and head of the hand meet a first preset condition and the hand belongs to the first hand shape; and / or detecting that the palm node and head of the hand meet a second preset condition and the hand belongs to the first hand shape.

[0059] For example, when the wrist node of the hand and the head meet the first preset condition, and / or the palm node of the hand and the head meet the second preset condition, it indicates that the palm of the hand faces the front of the head; the first hand shape is not a clenched fist shape.

[0060] For example, by analyzing the detected pose information, the system can determine whether the user's hand is in the first hand position. The first hand position refers to the preliminary posture that the user should adopt when preparing to perform an action or activate a function. In this scenario, the wrist node and head meet the first preset condition, and / or the palm node and head meet the second preset condition, and the first hand position is not a clenched fist (e.g., the palm is half-open or fully open). This provides the user with a natural starting point to prepare for the next interaction. When the first hand position is the preliminary posture, misidentification of subsequent hand positions can be avoided. The system needs to be able to recognize multiple first hand positions, including but not limited to release hand positions, thumbs-up hand positions, pinch hand positions, and any other hand positions, to adapt to different user habits and interaction needs.

[0061] For example, when the wrist node of the hand and the head meet a first preset condition, and / or the palm node of the hand and the head meet a second preset condition, it indicates that the palm is facing the front of the head, which can be understood as the palm or wrist must be facing the user's face. For example, this palm facing the front of the head can be called the head-hand mutual looking state.

[0062] For example, the clenched fist hand shape is used to describe a hand shape where the hand is clenched into a fist, with the fingers bent and close to or almost touching the palm. For example, see the clenched fist hand shape shown in Figure 2.

[0063] For example, the release hand shape can be a preset hand shape, which can be a hand shape that the user often uses when relatively relaxed, such as a hand shape with the palm open but the fingers not fully extended, or a hand shape with the palm fully open, etc. For example, the release hand shape shown in Figure 2 is a hand shape with the palm fully open.

[0064] Specifically, when the wrist node of the hand and the head meet the first preset condition and the hand belongs to the first hand shape, and / or the palm node of the hand and the head meet the second preset condition and the hand belongs to the first hand shape, it can be understood that the user enters the preparatory posture. The preparatory posture can, to a certain extent, satisfy the state of head and hand looking at each other (the wrist node of the hand and the head meet the first preset condition, and / or the palm node of the hand and the head meet the second preset condition).

[0065] For example, by detecting pose information to obtain the relative position and orientation of the head and hands, it can determine whether the user has entered a ready posture. This allows the system to prepare for specific interactive actions when the user enters a ready posture and provide corresponding visual feedback when conditions are met. Additionally, it prevents certain hand gestures and movements from triggering accidental operations when the user is not in a ready posture.

[0066] In some embodiments, detecting that the wrist node of the hand and the head meet a first preset condition includes: calculating a first connecting line between the wrist node and the head; and detecting that the angle between the wrist pose information of the wrist node and / or the head pose information of the head and the first connecting line meets the first preset condition.

[0067] For example, the positions of the wrist (i.e., the center point of the wrist in three-dimensional space) and the head (typically represented by the center of the head or the position of the eyes) are determined through image recognition (such as using deep learning algorithms to identify the positions of the wrist and head from a video stream) or through sensor data (such as an IMU or depth camera on a wearable device). Then, the straight-line distance between these two key points is calculated to form the first connecting line.

[0068] Next, the wrist pose information of the wrist node is detected. Wrist pose information may include the degree of wrist flexion, the wrist rotation angle, and the wrist's orientation. Wrist pose information can be obtained through sensor data or image recognition technology. Then, the wrist pose information is converted into a vector, and a first angle between the wrist pose vector and the first connecting line is calculated. For example, if the first angle is less than or equal to a first angle threshold set in a first preset condition (e.g., 30 degrees), then it is determined that the wrist pose information and the first connecting line satisfy the first preset condition.

[0069] For example, in addition to detecting wrist pose information, head pose information can be selectively detected. Head pose information typically includes head position (or orientation), tilt angle, etc. Head pose information can be obtained through sensor data or image recognition technology. Then, the head pose information is converted into a head pose vector, and a second angle between the head pose vector and the first connecting line is calculated. For example, if the second angle is less than or equal to a first angle threshold set in a first preset condition, then it is determined that the head pose information and the first connecting line satisfy the first preset condition.

[0070] If the first included angle and / or the second included angle meet the first angle threshold requirement in the first preset condition, then it is determined that the wrist node of the hand and the head meet the first preset condition.

[0071] For example, satisfying the first preset condition can be understood as satisfying the "hand-looking-head" state. For the "hand-looking-head" state, when viewing the head from the hand's perspective, the position of the wrist can be used as a reference point (wrist node). This helps determine the starting point of hand movements and serves as a stable reference in gesture recognition. For example, taking the first angle between the wrist pose vector and the first connecting line as an example, the first angle between the wrist pose vector and the first connecting line is calculated. This first angle reflects the degree of wrist tilt relative to the head, and the wrist pose vector can be the upward direction of the wrist (usually the upward direction perpendicular to the wrist node). When the first angle is less than or equal to a first angle threshold, it is determined that the first preset condition (satisfying the "hand-looking-head" state) is met.

[0072] The first preset condition ensures that the user's head and wrist nodes must be aligned to a certain extent. This means that the user's head orientation and wrist position need to be relatively consistent, reducing the possibility of accidental operation and providing the user with clear interactive feedback.

[0073] In some embodiments, detecting whether the palm node of the hand and the head meet a second preset condition includes: calculating a second connecting line between the palm node and the head, wherein the palm pose information of the palm node is determined based at least on the pose information of the first phalanx of one of the fingers of the hand and the wrist pose information of the wrist node of the hand; and detecting that the angle between the palm pose information of the palm node and / or the head pose information of the head and the second connecting line meets the second preset condition.

[0074] For example, using image recognition or sensor data, the positions of the palm node (i.e., the position of the center point of the palm in three-dimensional space) and the head (usually represented by the center of the head or the position of the eyes) are determined. Then, the straight-line distance between these two key points is calculated to form a second connecting line.

[0075] The palm pose information is typically determined based on the specific shape of the hand and the posture of the fingers. In one embodiment, the palm pose information can be calculated based on the pose information of the first phalanx of one of the fingers (e.g., the index finger) and the wrist pose information of the wrist node. The pose information of the first phalanx can be obtained through image recognition or sensor data, representing the finger's direction and degree of flexion. The wrist pose information can also be obtained through image recognition or sensor data, representing the degree of wrist flexion and the orientation of the palm. Combining these two pieces of information, the palm pose information (the overall pose of the palm) is calculated and converted into a palm pose vector. Then, the third angle between the palm pose vector and the second connecting line is calculated. For example, if the third angle is less than or equal to the second angle threshold (e.g., 30 degrees) set in the second preset condition, it is determined that the palm pose information and the second connecting line satisfy the second preset condition.

[0076] For example, a fourth angle between the head pose information and the second connecting line can be selectively detected. Head pose information typically includes the head's orientation and tilt angle. This head pose information can be obtained through sensor data or image recognition technology. Then, the head pose information is converted into a head pose vector, and the fourth angle between the head pose vector and the second connecting line is calculated. For example, if the fourth angle is less than or equal to a second angle threshold set in a second preset condition, then the head pose information and the second connecting line are determined to satisfy the second preset condition.

[0077] If the third included angle and / or the fourth included angle meet the second angle threshold requirement in the second preset condition, then it is determined that the palm node of the hand and the head meet the second preset condition.

[0078] For example, satisfying the second preset condition can be understood as satisfying the "head looking at hand" state. For the "head looking at hand" state, when considering the relative positions of the head and hand, the position of the palm is used as a reference point (palm node), because the palm is usually the most conspicuous and easily tracked part of the hand movements and gestures. For example, taking the fourth angle between the head pose vector and the second connecting line as an example, it is necessary to calculate the fourth angle between the head pose vector and the second connecting line. The fourth angle reflects the orientation of the head relative to the palm, and the head pose vector can be the forward direction of the head (usually the direction the user is facing). When the fourth angle is less than or equal to the second angle threshold, it is determined that the second preset condition (satisfying the "head looking at hand" state) is met.

[0079] The second preset condition ensures that the user's head and hand palm nodes must be aligned to a certain extent. This means that the user's head orientation and hand position need to be relatively consistent, reducing the possibility of accidental operation and providing the user with clear interactive feedback.

[0080] The palm pose information of the palm node is determined based on the pose information of the first phalanx of one of the fingers of the hand and the wrist pose information of the wrist node of the hand.

[0081] For example, the midpoint (0.5f) between the middle proximal joint of the middle finger and the wrist can be defined as the location of the palm. In this example, the middle proximal joint of the middle finger is chosen as a representative joint of the finger because the middle finger is located near the central area of ​​the palm in most hand shapes, and its dynamic changes can better reflect the overall movement and posture changes of the palm.

[0082] For example, a simplified hand geometry model can be built, the hand contour can be fitted using joint detection points, and then the geometric center of the hand geometry model can be calculated. This method can more accurately reflect the actual shape and position of the hand. Specifically, key points on the edge of the hand (such as the proximal ends of each finger, the edge of the wrist, etc.) are identified and marked, and then these key points are used to fit a polygonal or polyhedral hand geometry model. The center of the hand can be defined as the centroid of the hand geometry model or a representative point calculated by a specific algorithm.

[0083] For example, finger and wrist movements can be tracked in consecutive video frames, and time series analysis can be used to predict the optimal position of the palm node. This method is suitable for dynamic gesture recognition and real-time interaction scenarios.

[0084] In practical applications, the most suitable hand node determination strategy can be selected based on specific needs, computing resources, and accuracy requirements.

[0085] As shown in view (1) of Figure 3, when the user (target object) is detected to be in a ready posture, an interface element a can be displayed in the 3D environment as visual feedback of the ready posture. For example, the interface element a can be a circle. This UI element may be an interactive indicator that informs the user that the current ready posture has been recognized and is ready to receive further gesture commands.

[0086] In some embodiments, the method further includes determining the hand shape based on at least one of the values ​​of flexion angle, bending angle, and abduction angle of the finger joint of at least one finger of the hand.

[0087] For example, when detecting hand shape, it is necessary to accurately track the position of the ten joints of the five fingers and calculate at least one of the flexion, curvature, and abduction angles at each joint. This detailed detection ensures that the system accurately identifies the user's hand shape.

[0088] Figure 4 shows a schematic diagram of a hand, which includes the little finger (1), ring finger (2), middle finger (3), index finger (4), thumb (5), palm (6), and wrist (7). The palm (6) includes metacarpal bones (61) extending to the proximal ends of each finger. One end of each metacarpal bone (61) connects to the proximal end of each finger, and the other end connects to the wrist (7).

[0089] Each finger includes a fingertip, a distal phalanx, and a proximal phalanx. Except for the thumb (5), the other fingers also include the intermediate phalanx.

[0090] The little finger 1 includes the little fingertip 11, the little finger distal 12, the little finger intermediate 13, and the little finger proximal 14.

[0091] Among them, the ring finger 2 (Ring) includes the ring fingertip 21 (Ring Tip), the ring finger distal 22 (Ring Distal), the ring finger intermediate 23 (Ring Intermediate), and the ring finger proximal 24 (Ring Proximal).

[0092] Among them, the middle finger 3 (Middle) includes the middle fingertip 31 (Middle Tip), the middle finger distal 32 (Middle Distal), the middle finger intermediate 33 (Middle Intermediate), and the middle finger proximal 34 (Middle Proximal).

[0093] The index finger 4 includes the index tip 41, the index distal 42, the index intermediate 43, and the index proximal 44.

[0094] The thumb (5) includes the thumb tip (51), the thumb distal (52), and the thumb proximal (53).

[0095] The flexion angle typically refers to the bending angle between the palm and finger joints. In hand shape detection, the flexion angle is usually used to measure the degree of bending of the fingers towards the palm.

[0096] As shown in Figure 5, for thumb flexion, when the flexion angle θ is greater than 155°, the finger posture is Open, indicating that the finger is straight / open; when the flexion angle θ is less than 120°, the finger posture is Closed, indicating that the finger is bent / closed. For four-finger flexion, when the flexion angle θ of any of the four fingers is greater than 144°, the finger posture is Open, indicating that the finger is straight / open; when the flexion angle θ is less than 126°, the finger posture is Closed, indicating that the finger is bent / closed.

[0097] Please continue to refer to Figure 4. Taking the flexion angle β of the index finger 4 as an example, the proximal end 44 of the index finger is the vertex B of the flexion angle β, the line connecting the proximal end 44 of the index finger to the middle phalanx 43 of the index finger is the first side BC of the flexion angle β, and the metacarpal bone 61 connected to the proximal end 44 of the index finger is the second side BA of the flexion angle β.

[0098] Among them, "curl" usually refers to the bending angle of the distal phalanx of the finger.

[0099] As shown in Figure 6, for thumb bending, when the bending angle θ is greater than 90°, the finger posture is Open, indicating that the finger is straight / open; when the bending angle θ is less than 90°, the finger posture is Closed, indicating that the finger is bent / closed. For four-finger bending, when the bending angle θ of any of the four fingers is greater than 107°, the finger posture is Open, indicating that the finger is straight / open; when the bending angle θ is less than 73°, the finger posture is Closed, indicating that the finger is bent / closed.

[0100] Please continue to refer to Figure 4. Taking the bending angle γ of the index finger 4 as an example, the extension of the line ED connecting the fingertip 41 to the distal end 42 of the index finger is the first side of the bending angle γ, and the extension of the line BC connecting the proximal end 44 to the middle phalanx 43 of the index finger is the second side of the bending angle γ. The intersection of the extension of the line ED and the extension of the line BC is the vertex of the bending angle γ.

[0101] The abduction angle typically refers to the angle between two adjacent fingers. For example, starting with the thumb, the abduction angles between the current finger and the next finger include: the abduction angle between the thumb and index finger, the abduction angle between the index finger and middle finger, the abduction angle between the middle finger and ring finger, and the abduction angle between the ring finger and little finger.

[0102] As shown in Figure 7, for thumb abduction, when the abduction angle θ is greater than 13°, the finger posture is Open, indicating that the finger is spread out; when the abduction angle θ is less than 13°, the finger posture is Closed, indicating that the finger is closed. For four-finger abduction, when the abduction angle θ of any of the four fingers is greater than 10°, the finger posture is Open, indicating that the finger is spread out; when the abduction angle θ is less than 10°, the finger posture is Closed, indicating that the finger is closed.

[0103] Please continue to refer to Figure 4. Taking the abduction angle α between the middle finger 3 and the index finger 4 as an example, the extension of the line GF connecting the middle phalanx 33 of the middle finger to the proximal end 34 of the middle finger is the first side of the abduction angle α, and the extension of the line CB connecting the middle phalanx 43 of the index finger to the proximal end 44 of the index finger is the second side of the abduction angle α. The intersection of the extension of the line GF and the extension of the line CB is the vertex of the abduction angle α.

[0104] For example, in the second hand position (clenched fist), the joints of the hand (at least four fingers) bend towards the palm, and the fingers are close together. When clenching a fist, the flexion angle (e.g., 90°), bending angle (e.g., 60°), and abduction angle (e.g., 2°) are all relatively small.

[0105] For example, in the first hand position, which is the release hand position, the fingers are fully extended and no longer bent, but the abduction angle may remain unchanged or increase slightly. For instance, in the release hand position, the flexion angle of the finger joints is close to 180°, the bending angle is also close to 180°, and the abduction angle is greater than 10°.

[0106] For example, in a head-hand interaction state (where the wrist joint of the hand and the head meet a first preset condition, and / or the palm joint of the hand and the head meet a second preset condition), the system determines whether the hand shape is the first hand shape (not a clenched fist) by detecting the flexion angle, bending angle, and abduction angle of the finger joints. If the hand shape meets the conditions for the first hand shape, the user can proceed to the preparatory posture.

[0107] For example, if the user's hand gesture is already in the second hand position (clenched fist) before the head-and-hand eye contact state, the system considers this state as locked, and the user cannot directly enter the preparatory posture process. The user needs to exit the second hand position, end the locked state, and then the system re-detects the preparatory posture. Only when the preparatory posture conditions are met can the user enter the preparatory posture process.

[0108] Step 120: In response to the release of the second hand shape in the target pose, the target function corresponding to the target pose is triggered.

[0109] In some embodiments, the step of triggering the target function corresponding to the target pose in response to the release of the second hand shape in the target pose includes: if the second hand shape is released in the first pose, then triggering the first target function corresponding to the first pose; or if the second hand shape is released in the second pose, then triggering the second target function corresponding to the second pose.

[0110] The hand pose can include the hand's position and / or posture information. When the hand is in the target pose and is released with a second hand shape, the target function corresponding to the target pose is triggered.

[0111] The first pose and the second pose can have different postures or different positions, or they can have different positions and postures.

[0112] There are multiple ways to represent hand pose. For example, the position and / or posture information of the wrist node can be selected as the position and / or posture information of the hand. For instance, the wrist pose information of the wrist node can be obtained. The wrist pose information includes at least the wrist rotation angle, and can also include the position and / or posture information of the wrist. Then, the hand pose can be determined based on the wrist pose information of the wrist node.

[0113] In some embodiments, the first posture is with the palm of the hand facing the front of the head; or the second posture is with the back of the hand facing the front of the head.

[0114] In some embodiments, the first target function is a homepage access function; or the second target function is a reset center function.

[0115] Specifically, when the target object is in a ready posture and the hand is determined to switch from the first hand shape (not a clenched fist) to the second hand shape (a clenched fist), the function selection event is activated. The function selection event selects different functions based on the different hand postures.

[0116] As shown in view (1) of Figure 3, when the user (target object) is detected to be in a preparatory posture, an interface element a can be displayed in the three-dimensional environment as visual feedback of the preparatory posture. For example, the interface element a can be a circle icon.

[0117] As shown in view (2) of Figure 3, when the user (target object) is detected to be in a ready posture and the hand shape is switched from the first hand shape (not a fist) to the second hand shape (fist), the function selection event is activated.

[0118] Once the hand position is switched from the first hand shape to the second hand shape to activate the function selection event, different postures can be switched by rotating the wrist to select the home access function (Home function) or the reset center function (Recenter function).

[0119] In some embodiments, the method further includes: determining the pose of the second hand shape based on the wrist pose information of the wrist node of the hand.

[0120] The wrist pose information includes at least the wrist rotation angle. The first pose is with the palm facing the head; the second pose is with the back of the hand facing the head. For example, wrist pose information is acquired using sensors or image recognition technology. The wrist pose information includes at least the wrist rotation angle, and may also include wrist position and / or posture information. Then, based on the wrist pose information, particularly the wrist rotation angle, the pose of the second hand is determined. For example, if the wrist rotation angle causes the palm to face the head, the second hand is in the first pose; if the wrist rotation angle causes the back of the hand to face the head, the second hand is in the second pose. This determination can be based on a pre-set threshold or range, or on the prediction results of a machine learning model.

[0121] In some embodiments, the method further includes: dynamically adjusting the weight of at least one of the flexion angle, bending angle and abduction angle based on the wrist posture information of the hand.

[0122] For example, a weighted mapping function can be pre-defined or learned, which maps wrist pose information to weighted values ​​for flexion, bending, and abduction angles. This function can be based on physical simulation, machine learning, or a hybrid model that combines the advantages of both.

[0123] Then, based on the current wrist pose information, the weight mapping function calculates the weight values ​​corresponding to each angle. These weight values ​​are used to adjust the importance and influence of each angle when calculating hand joint angles. For example, if the wrist is in a certain rotational state, more attention may need to be paid to changes in the abduction angle, so the weight of the abduction angle will be increased accordingly. For example, when the wrist rotates in a certain direction, the weight of the flexion angle of the relevant finger joints in that direction may need to be increased to better simulate the natural posture of the hand following the wrist movement.

[0124] After adjusting the weights, the angles of the hand joints will be recalculated based on the new weight values ​​to optimize hand posture.

[0125] For example, the weight of at least one of the values ​​of flexion angle, bending angle and abduction angle can be dynamically adjusted based on the wrist rotation angle of the hand.

[0126] For example, the greater the wrist rotation angle, the smaller the weight of the flexion angle and the bending angle.

[0127] In particular, considering the occlusion problem in gesture tracking, since the wrist rotation is obstructed by the fingers, the detection of flexion, curl and abduction angles is inaccurate. Therefore, the weight of at least one of the flexion, curl and abduction angles is dynamically adjusted according to the wrist rotation angle of the hand.

[0128] For example, when the second hand position is rotated to the second pose to select the Recenter function, the flexion, curl, and abduction angles can be disregarded when the hand is in a clenched fist position corresponding to the Recenter function. Specifically, the larger the wrist rotation angle corresponding to the second hand position (the closer it is to the Recenter function), the lower the weight of the flexion, curl, and abduction angles; conversely, the smaller the wrist rotation angle (the closer it is to the Home function), the higher the weight of these angles.

[0129] For example, the detection parameters for wrist rotation can be adjusted. When the degree of the first angle corresponding to the "hand looking at head" state is less than the third angle threshold (e.g., 30°), the detection parameter for wrist rotation is 0; when the degree of the first angle corresponding to the "hand looking at head" state is greater than the third angle threshold (e.g., 90°), the detection parameter for wrist rotation is 1.

[0130] In some embodiments, the method further includes: displaying a function icon corresponding to the target function in response to the second hand being in a target pose.

[0131] As shown in view (2) of Figure 3, if the target pose is the first pose, which means that the palm of the hand is facing the front of the head, then the function is switched according to the first pose of the second hand shape, so that the first target function corresponding to the first pose is the home page access function (Home function), and the first function icon b corresponding to the home page access function is displayed in the three-dimensional environment. For example, the first function icon b can be a house icon. For example, the first function icon b can be displayed in any position around the hand, such as above the hand, or in front of the palm, etc.

[0132] As shown in view (4) of Figure 3, if the target pose is the second pose, which means that the back of the hand is facing the front of the head, then the function is switched according to the second pose of the second hand shape, so that the second target function corresponding to the second pose is the Recenter function, and the second function icon d corresponding to the Recenter function is displayed in the three-dimensional environment. For example, the second function icon d can be a crosshair icon. For example, the second function icon d can be displayed at any position around the hand, such as above the hand, or behind the hand, etc.

[0133] In some embodiments, the method further includes: displaying a user interface corresponding to the target function.

[0134] For example, displaying the user interface corresponding to the target function within a three-dimensional environment.

[0135] For example, after selecting the homepage access function based on the first pose shown in view (2) of Figure 3, when the second hand shape is detected to be released in the first pose, that is, when the hand shape is detected to switch from the clenched fist shape shown in view (2) to the released hand shape shown in view (3), the homepage access function is triggered, and the first user interface c corresponding to the homepage access function is displayed in the three-dimensional environment.

[0136] For example, after selecting the reset center function based on the second pose shown in view (4) in Figure 3, when the second hand shape is detected to be released in the second pose, that is, when the hand shape is detected to switch from the clenched fist shape shown in view (4) to the released hand shape shown in view (5), the reset center function is triggered, and the second user interface e corresponding to the reset center function is displayed in the three-dimensional environment.

[0137] In some embodiments, the duration of the second hand shape before release is less than a first time threshold.

[0138] For example, the duration of the second hand gesture before release represents the duration of the clenched fist gesture. For example, the first time threshold is 5 seconds. If the duration is less than the first time threshold, the user is considered to be performing a continuous operation.

[0139] In some embodiments, the method further includes: canceling the triggering of the target function in response to the second hand type being maintained for a period of time before release exceeding a first time threshold.

[0140] For example, if the second hand shape (clenched fist shape) is maintained for more than the first time threshold (e.g., 5 seconds) before release, the target function will be canceled.

[0141] In some embodiments, the second hand shape being released at the target pose includes: the hand exiting the second hand shape at the target pose and switching to the release hand shape within a second time threshold.

[0142] In some embodiments, the method further includes: canceling the triggering of the target function in response to the hand exiting the second hand shape and not switching to the release hand shape within a second time threshold.

[0143] For example, if the duration of the clenched fist (second hand position) is less than a first time threshold (e.g., 5 seconds), when the hand is detected to switch from a clenched fist to a released hand position in the target pose, and the hand position switching time is less than a second time threshold (e.g., 300 milliseconds), the target function is triggered. The first time threshold is greater than the second time threshold.

[0144] For example, "exiting the clenched fist" describes a situation where the hand is no longer recognized as a clenched fist. It may be recognized as another hand shape or it may not be recognized as any hand shape, but it is no longer a clenched fist. For example, as shown in Figure 2, the hand is half-open and the fingers are naturally curved when exiting the clenched fist.

[0145] The system detects whether the hand is in a released fist position upon triggering. If it is, a transition phase begins. During this phase, it checks whether the hand has switched to a released hand position within a second time threshold (e.g., 300 milliseconds). If the hand switches to a released hand position within the second time threshold (e.g., 300 milliseconds), the function corresponding to the wrist orientation is triggered. If the transition phase exceeds the second time threshold (e.g., 300 milliseconds), the triggering of the target function is canceled.

[0146] For example, if the clenched fist gesture is maintained for more than a first time threshold (e.g., 5 seconds), it indicates that the user has no intention of triggering the function, and the function selection event is canceled and / or the target function is canceled.

[0147] For example, if the duration of the clenched fist hand position is less than a first time threshold (e.g., 5 seconds), when the hand is detected to have switched from the clenched fist hand position to the unclenched fist hand position in the target position, but has not switched from the unclenched fist hand position to the released hand position, the function selection event is canceled and / or the target function is canceled.

[0148] For example, if the duration of the clenched fist hand position is less than the first time threshold (e.g., 5 seconds), when it is detected that the hand in the target position changes from the clenched fist hand position to the released hand position, and the hand position switching time exceeds the second time threshold (e.g., 300 milliseconds), that is, the transition phase exceeds the second time threshold (e.g., 300 milliseconds), the target function is canceled.

[0149] In some embodiments, the method further includes: hiding the function icon when the target function is canceled.

[0150] For example, hiding the icon corresponding to the target function is an intuitive and effective feedback method when a function selection event is canceled and / or the target function is deactivated. This helps reduce interface clutter, making other functions stand out more, thereby improving the user experience. The specific implementation of hiding function icons may vary depending on the application or device.

[0151] For example, the fade-out animation can hide the function by gradually fading out when the function selection event is canceled, until it disappears completely. This fade-out animation effect not only provides visual feedback but also adds to the fun of the interaction.

[0152] For example, the swipe-to-hide method: In some cases, the function icon corresponding to the target function can be swiped to the edge or bottom of the screen to be hidden. This method is often used in interfaces with multiple scrollable pages.

[0153] Hiding by disappearing directly: In some minimalist applications, the function icon corresponding to the target function may disappear directly without any animation.

[0154] By hiding the corresponding function icon when canceling a function selection event and / or canceling the triggering of a target function, applications or devices can provide users with a clearer and cleaner interface. This helps reduce user confusion and accidental operations during use, thereby improving the overall user experience.

[0155] For example, hiding feature icons can also serve as a visual cue, informing users that the feature is currently unavailable or has been canceled. This is especially important for novice users or those less familiar with the application.

[0156] For example, in addition to hiding function icons, other forms of feedback (such as sound prompts, vibration feedback, etc.) could be considered to meet the needs of different users.

[0157] In some embodiments, the method further includes: determining a triggering area corresponding to the target function before triggering the target function, wherein the range of the triggering area is determined based on the initial palm pose when the hand is switched to the second hand shape; and prohibiting the triggering of the target function when the hand is detected to be outside the triggering area.

[0158] For example, before triggering a target function, it is necessary to determine the corresponding trigger area. This trigger area is determined based on the initial palm pose when the hand switches to the second hand position. For instance, when a user switches their hand from a first pose (palm facing head) to a second pose (back of hand facing head), a trigger area is determined based on the current palm position (i.e., the initial palm pose) and preset rules. This trigger area can be two-dimensional (such as a projection on the user interface) or three-dimensional (in three-dimensional space), depending on the application scenario and requirements.

[0159] After determining the trigger area, the hand position is detected. If the hand remains within the trigger area and other trigger conditions (such as hand posture, gesture, etc.) are met, the corresponding target function will be triggered. However, if the hand is detected to be outside the trigger area, the target function will be prevented from being triggered regardless of whether other conditions are met. This design prevents users from accidentally triggering unwanted functions.

[0160] In some embodiments, the method further includes displaying a visual indication of the trigger area.

[0161] In some embodiments, the method further includes: adjusting the size of the trigger area according to the moving speed of the second hand away from the initial center point position of the trigger area.

[0162] For example, when a user activates a function selection event by clenching their fist, a visual indicator of the trigger area can be displayed within the 3D environment. This indicator appears when the user's head is within a suitable distance from their hand. This visual indicator can be a semi-transparent bounding box, a highlighted area, simple lines, shapes, color variations, or any other graphic element that clearly conveys location information. By displaying this visual indicator, users can more intuitively understand the location and size of the trigger area, making it easier to trigger the target function. Furthermore, the display method and parameters of the visual indicator can be dynamically adjusted based on user preferences and habits.

[0163] For example, the trigger area is a circular region, with its initial center point positioned at the initial palm position when the hand switches to the second hand shape (i.e., the palm position when the user makes a fist). The initial size (radius) of the trigger area is a preset radius value (e.g., 10cm). This design helps users intuitively understand which areas are effective interaction zones.

[0164] After determining the initial range of the trigger area, the system monitors the speed of the user's hand movement in real time. If the user's hand, based on a secondary hand shape, quickly moves away from the initial center point, the system assumes the user may not intend to interact with the function and therefore dynamically reduces the size of the trigger area. This design increases the system's flexibility and user experience because it allows the system to adjust the sensitivity of the interaction based on the user's actual behavior.

[0165] If the system detects that the user's hand, based on a second hand shape, has completely exceeded the trigger area, then the corresponding target function will be disabled. This design prevents users from unintentionally triggering certain functions, thereby improving the stability and security of the system.

[0166] In some embodiments, when the first relative distance between the head and the hand exceeds a first distance threshold, the visual indication of the trigger area is hidden.

[0167] In addition to the speed and position of hand movements, the system also considers the initial relative distance between the user's head and hand. If this initial relative distance exceeds a certain threshold (e.g., the user's hand moves out of their field of vision), the system will hide the visual indicator in the trigger area. This design helps the system manage the user interface more intelligently and reduces unnecessary visual distractions.

[0168] As shown in Figure 8, taking the activation of the Home access function as an example, the user can select the Home access function using a second hand shape (a clenched fist). The initial center point O is set with the initial palm position when the hand switches to the second hand shape (i.e., the palm position when the user clenches their fist), and the default effective range is a preset radius value (e.g., 10cm). Figure 8 also shows the first function icon b corresponding to the Home access function.

[0169] As shown in Figure 9, taking the activation of the homepage access function (Home function) as an example, if the user's hand moves away from the initial center point O, and the movement speed exceeds the speed threshold, it is judged as rapid movement, and it is considered that the user has no intention to perform functional interaction. Therefore, the trigger area S corresponding to the homepage access function is quickly reduced to the adjusted trigger area S' shown in Figure 9. Figure 9 also shows the first function icon b corresponding to this homepage access function.

[0170] In order to ensure that the effective range of the trigger area is within the user's comfort range, the trigger area will not be effective if the hand position is too high or too low.

[0171] For example, if the hand is positioned above the horizontal level of the head (or the augmented reality device or head-mounted display), the trigger area will not function. This is because a hand position that is too high may require the user to look up or overextend their arms, increasing fatigue and discomfort. Furthermore, a hand position that is too high may also exceed the system's tracking range, leading to inaccurate positioning or loss of tracking.

[0172] For example, if the wrist is below the ground level in a 3D environment, the trigger area will not be active. This is because a hand position that is too low may require the user to bend over or excessively bend their arm, which will increase fatigue and discomfort. In addition, a hand position that is too low may be affected by interference from the ground or other obstacles, affecting the tracking accuracy and stability of the system.

[0173] In some embodiments, adjusting the size of the trigger area based on the moving speed of the second hand away from the initial center point of the trigger area includes: when the moving speed of the second hand away from the initial center point exceeds a speed threshold, the faster the moving speed, the smaller the trigger area; or when the moving speed of the second hand away from the initial center point does not exceed a speed threshold, the trigger area maintains the initial area.

[0174] For example, a speed threshold needs to be set, which is usually determined based on multiple experiments and user feedback to ensure a suitable interactive experience in different scenarios. The speed threshold is designed to distinguish between two different user intentions: fast movement and slow movement.

[0175] For example, when the system detects that the second hand is moving away from the initial center point at a speed exceeding a speed threshold, which is considered rapid movement, the system will dynamically adjust the size of the trigger area based on the movement speed. The rationale behind this design is that when the second hand moves quickly, the user may want more precise control over the interaction; therefore, reducing the size of the trigger area can improve the accuracy of the interaction.

[0176] For example, a mapping function can be set up to correspond to the movement speed and the trigger area, calculating the corresponding trigger area size based on the movement speed of the second hand. This function can be linear or non-linear, depending on the design requirements.

[0177] For example, the relationship between the movement speed v and the radius r of the trigger area can be represented by the following linear function: r = r_initial - k*(v - v_threshold);

[0178] Where r_initial is the initial radius of the trigger region; k is a constant representing the rate at which the radius decreases as the velocity increases (it can be a positive number); and v_threshold is the velocity threshold. When v exceeds v_threshold, the radius r of the trigger region decreases linearly as v increases.

[0179] For example, the relationship between the movement speed v and the radius r of the trigger area can be represented by a nonlinear mapping function, which allows for more flexible adjustment of the size of the trigger area. For instance, quadratic, exponential, or logarithmic functions can be used.

[0180] For example, the relationship between the movement speed v and the radius r of the trigger area can be expressed by the following quadratic function: r = r_initial - k*(v - v_threshold)2;

[0181] In this function, when v slightly exceeds v_threshold, the rate of decrease of r is relatively slow; however, as v increases further, the rate of decrease of r increases rapidly. This allows for a smoother adjustment of the trigger area size to accommodate interaction requirements at different speeds.

[0182] For example, during the adjustment of the trigger area size, user feedback can be incorporated to further optimize the mapping function. For instance, if the second hand gesture is frequently mis-triggered at a certain speed, the system can appropriately reduce the trigger area size corresponding to that speed.

[0183] For example, if the movement speed of the second hand away from the initial center point does not exceed the speed threshold, it is considered slow movement. The system assumes the user may be performing a more detailed interaction or requires more space. Therefore, in this case, the trigger area will maintain its initial size to ensure the user has sufficient space for interaction.

[0184] In some embodiments, the method further includes: adjusting the display of the function icon based on the direction of movement of the hand and / or a first relative distance between the head and the hand.

[0185] For example, when a user's hand is detected moving in a specific direction, the display position of the function icon can be adjusted accordingly to maintain a natural alignment with the hand or the optimal viewing angle. For instance, if the user is moving their hand to the right, the relevant function icon will also smoothly move to the right, ensuring that the user can clearly see and easily reach the icon without having to turn their head or hand significantly. This following mechanism significantly enhances the intuitiveness and responsiveness of the interaction, reducing the time users spend searching for and locating functions in three-dimensional space.

[0186] For example, in addition to considering the direction of hand movement, the system also detects the initial relative distance between the head and hand. An increase in this initial relative distance indicates that the user may be performing an operation at a greater distance or seeking a wider field of view. In this case, not only will the display position of the function icons be adjusted based on the hand position, but the visual effects of the function icons may also be appropriately adjusted, such as enlarging the size or increasing the brightness of the function icons, to ensure clear visibility even at a greater distance. Conversely, when the initial relative distance between the head and hand decreases, the function icons can be restored to their standard size or adjusted to a state more suitable for close-up viewing, avoiding visual interference or excessively strong light stimulation.

[0187] By combining the direction of hand movement and the initial relative distance between the head and hands, the display position and shape of function icons can be adaptively adjusted, creating a personalized interactive interface that optimizes according to changes in user actions and perspective. This not only reduces the cognitive load on users navigating in three-dimensional space but also greatly improves the naturalness and efficiency of interaction, allowing users to operate in virtual or augmented reality environments as freely and intuitively as in the real world.

[0188] In some embodiments, adjusting the display of the function icon based on the direction of hand movement and / or the first relative distance between the head and the hand includes:

[0189] If the movement direction is horizontal, an interpolation calculation is performed based on a preset weighting factor, the initial display position of the function icon, and the current palm position of the hand to obtain a first interpolation result, and the display position of the function icon is adjusted according to the first interpolation result; or

[0190] If the movement direction is vertical and the first relative distance is greater than the second relative distance, then an interpolation result is obtained by interpolating based on a preset weighting factor, the initial display position of the function icon, and the current palm position of the hand. The display position of the function icon is then adjusted according to the second interpolation result, wherein the second relative distance is the distance between the initial display position and the head; or

[0191] If the direction of movement is vertical and the first relative distance is less than the second relative distance, then the display position of the function icon is controlled to maintain a fixed distance from the head in the vertical direction;

[0192] Wherein, the longitudinal direction is the direction of movement back and forth along the line connecting the head and the hand; the lateral direction is the direction of movement left and right along the line perpendicular to the connecting line.

[0193] For example, as shown in view (1) of Figure 10, taking the activation of the home access function as an example, the first function icon b corresponding to the home access function is displayed above the palm of the hand, and the initial display position P of the function icon is projected on the hand and coincides with the initial center point position O of the trigger area S.

[0194] For example, as shown in view (2) of Figure 10, when the hand is detected to be moving laterally (i.e., moving left and right along the vertical line connecting the head and hand), interpolation can be performed based on a preset weighting factor (e.g., 0.3), the initial display position P, and the current palm position Q of the hand. This interpolation is typically used to smoothly move the position of the function icon so that it follows the lateral movement of the hand. Specifically, the relative positional relationship between the current palm position Q and the initial display position P is first calculated. Then, based on the preset weighting factor (which can be adjusted according to application requirements or user preferences), interpolation is performed on these two positions to obtain a new display position P' between them, i.e., the first interpolation result. Then, the display position of the function icon b is adjusted according to the first interpolation result so that the function icon b follows the lateral movement of the hand in a spring-like motion, visually appearing as a function icon b that is difficult to drag, to prompt the user to trigger the homepage access function corresponding to the function icon b as soon as possible.

[0195] For example, as shown in view (1) of Figure 11, taking the activation of the home access function as an example, the first function icon b corresponding to the home access function is displayed above the palm of the hand. The initial display position P of the first function icon b is projected onto the hand and coincides with the initial center point position O of the trigger area S (not shown in the figure). When the movement direction of the hand is detected to be longitudinal (i.e., the direction of movement along the line connecting the head and the hand), the adjustment strategy will be slightly more complicated because it also needs to consider the relative distance between the head and the hand.

[0196] For example, when the system detects that the hand is moving vertically (i.e., moving back and forth along the line connecting the head and hand), if the first relative distance (the current distance between the head and hand) is greater than the second relative distance (the distance between the initial display position and the head), it indicates that the hand is moving away from the head. In this case, the system will also use interpolation to calculate the second interpolation result based on a preset weighting factor, the initial display position, and the current palm position. Then, based on this second interpolation result, the system will adjust the display position of the function icon so that the function icon moves in a spring-like motion following the vertical movement of the hand.

[0197] For example, as shown in view (2) of Figure 11, when the movement direction of the hand is detected to be vertical (i.e., moving back and forth along the line connecting the head and the hand), if the first relative distance (i.e., the current distance between the head and the hand) is less than the second relative distance (i.e., the distance between the initial display position and the head), it indicates that the hand is moving closer to the head. In this case, in order to avoid the first function icon b obstructing the user's view or causing misoperation, the system will control the display position of function icon b to maintain a fixed distance from the head in the vertical direction.

[0198] In some embodiments, the method further includes: canceling the triggering of the target function in response to the second hand being in a target pose and the descent distance exceeding a second distance threshold.

[0199] First, the system needs to be able to recognize the user's second hand gesture and determine whether that second hand gesture is in a target pose (such as a first pose or a second pose) to trigger a specific target function. For example, the system may begin detecting changes in the position of the second hand gesture before the target pose is released. In particular, it may detect changes in the vertical direction (such as up and down movement). This detection of position changes is typically achieved using some form of position sensing technology (such as camera tracking, touchscreen sensing, or sensors in wearable devices).

[0200] For example, a second distance threshold can be set. This second distance threshold is set based on the application scenario and user needs, and is usually determined based on the normal range of vertical hand movement. If the downward distance of the hand in the vertical direction exceeds the second distance threshold, it is assumed that the user may no longer intend to perform the previous interaction, and the target function is canceled.

[0201] For example, the second distance threshold can be smaller than the first distance threshold.

[0202] After canceling the triggered target function, you can also provide some feedback to the user to confirm that the triggering event has been canceled. This feedback can be visual (such as messages or icons on the screen), auditory (such as sound prompts), or tactile (such as vibration prompts).

[0203] In some embodiments, if the second hand shape is released in the first pose, before triggering the first target function corresponding to the first pose, the method further includes: canceling the triggering of the first target function when the hand of the released hand shape moves away from the initial center point position of the trigger area by more than a third distance threshold.

[0204] For example, if the second hand gesture is released from the first gesture to switch to the release gesture, and the hand quickly moves away from the initial center point of the trigger area, exceeding a preset third distance threshold (e.g., 5cm), the system will assume the user is making a mistake or adjusting their hand position, rather than genuinely intending to trigger the primary target function (such as the Home function). Even if the primary target function (such as the Home function) is activated, it will be automatically canceled during movement or dragging. Therefore, in this case, the system will cancel the triggering of the primary target function, thus avoiding unnecessary misoperations. By combining the distance of hand movement to determine whether a target function is triggered, the system can more accurately understand the user's intent and reduce the possibility of misoperations.

[0205] For example, the third distance threshold can be less than the second distance threshold.

[0206] Please refer to view (2) in Figure 10, where the initial display position P of the function icon is projected onto the hand and coincides with the initial center point O of the trigger area S. The hand displacement distance can be determined based on the initial center point O of the trigger area S and the current palm position Q of the released hand shape. If the hand displacement distance exceeds the third distance threshold (e.g., 5cm), the first target function is canceled. If the hand displacement distance does not exceed the third distance threshold (e.g., 5cm), the first target function is triggered.

[0207] In some embodiments, if the second hand shape is released in the first pose, before triggering the first target function corresponding to the first pose, the method further includes: triggering the first target function in response to the maintenance time of the released hand shape exceeding a third time threshold; and / or canceling the triggering of the first target function in response to the maintenance time of the released hand shape not exceeding the third time threshold.

[0208] In some embodiments, the method further includes: displaying visual and / or auditory indications corresponding to the duration of the release hand gesture.

[0209] For example, after entering the first target function (such as the Home function), a stronger intention is needed to determine the release to trigger the first target function. After entering the first target function and releasing the second hand gesture, the release hand gesture needs to be maintained until the third time threshold before the first target function is triggered.

[0210] For example, the third time threshold can be smaller than the second time threshold.

[0211] For example, this third time threshold can be less than the duration of a long press operation in the system (e.g., within 250 milliseconds).

[0212] By setting a third time threshold (e.g., 250 milliseconds), the system requires users to maintain a specific hand gesture for a certain period before executing the corresponding function. This approach effectively reduces accidental touches and increases the certainty and purposefulness of user operations. While the user releases the hand gesture and waits for the function to trigger, visual cues (such as progress bars, countdown animations, color changes, numerical displays, highlighted states, and charging effect animations) and / or auditory cues (such as sound effects and voice prompts) can be used to indicate the progress of the gesture, allowing the user to clearly perceive the operation status and enhancing the immediate feedback mechanism of the interaction. This is crucial for improving the user's sense of control and trust in the system.

[0213] For example, the following are some examples of auditory instructions:

[0214] Progress Bar: A horizontal or vertical progress bar is displayed on the interface. When the user releases their hand gesture, the progress bar begins to gradually fill from left to right (or from 0% to 100%). The filling speed can be adjusted according to actual needs. When the third time threshold (e.g., 250 milliseconds) is reached, the progress bar is completely filled, possibly accompanied by a color change or animation effect, indicating that the function has been triggered.

[0215] Countdown Animation: A countdown timer is displayed on the interface, starting from "250ms" and decreasing to "0". During the countdown, the number can change dynamically, accompanied by slight flashing or color gradient effects to attract the user's attention. When the countdown ends, the number disappears or a message such as "Trial Successful" appears.

[0216] Color Change: After the user releases their hand gesture, the color of the relevant interface elements (such as the trigger area, function icons, etc.) begins to change, gradually transitioning from the initial color to another color. The speed of the color change corresponds to the duration of the change. When the third time threshold (such as 250 milliseconds) is reached, the color changes completely, indicating that the function has been activated.

[0217] Numerical display: A number is displayed directly on the interface to indicate the time the user has maintained the hand gesture (e.g., "150ms / 200ms / 250ms"). The number updates in real time as the time progresses until it reaches a threshold and stops changing or displays a specific prompt message.

[0218] Highlighted State: The interface element corresponding to the released hand gesture (such as a function icon, the edge of the trigger area, etc.) becomes highlighted for the duration of the gesture. The highlight color can be user-defined or system preset. The highlighted state lasts until the function is triggered, after which it may return to the normal state.

[0219] Charging-up animation: Simulates the charging-up process, such as the energy ball gradually growing larger and the light gradually converging. After the user releases their hand gesture, the animation begins to play. As the duration increases, the animation effect gradually intensifies until it reaches a threshold, at which point the animation peaks, indicating that the function is ready to be triggered.

[0220] For example, the following are some examples of visual cues:

[0221] Sound effect prompts: When the user releases their hand and starts the timer, a soft background sound effect (such as an electrical noise or wind sound) is played. As the duration increases, the sound effect gradually intensifies or changes its rhythm. When the threshold is reached, a confirmation sound effect (such as a "ding" sound or a mechanical start-up sound) is played to give the user clear feedback.

[0222] Voice prompts: The system announces the current status to the user via voice, such as "Preparing to trigger the function, please keep your hand shape unchanged" or "Trigger conditions met, the function will be executed soon." Voice prompts can be customized according to the user's language preferences, increasing the personalization of the interaction.

[0223] All the above-mentioned technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0224] This embodiment of the disclosure determines when the hand changes from a first hand shape to a second hand shape; in response to the release of the second hand shape at a target pose, it triggers the target function corresponding to the target pose. This embodiment of the disclosure achieves a natural interaction method by recognizing the user's hand switching from one specific hand shape (first hand shape) to another (second hand shape), and releasing the second hand shape at a specific target pose, thereby triggering the function associated with that target pose. This simplifies the interaction process; the user does not need to remember complex instructions or operation sequences, but can trigger functions simply by changing hand posture, reducing the user's memory burden and improving interaction efficiency.

[0225] To facilitate better implementation of the interaction method of the embodiments of this disclosure, the embodiments of this disclosure also provide an interaction device. Please refer to FIG12, which is a schematic structural diagram of the interaction device provided in the embodiments of this disclosure. The interaction device 200 may include:

[0226] The determining unit 210 is used to determine whether the hand is switched from a first hand shape to a second hand shape;

[0227] The interaction unit 220 is used to trigger the target function corresponding to the target pose in response to the release of the second hand shape in the target pose.

[0228] In some embodiments, the interaction unit 220 can be used to: if the second hand shape is released in the first pose, trigger the first target function corresponding to the first pose; or if the second hand shape is released in the second pose, trigger the second target function corresponding to the second pose.

[0229] In some embodiments, the interaction unit 220 can also be used to display the user interaction interface corresponding to the target function.

[0230] In some embodiments, before determining that the hand has switched from the first hand shape to the second hand shape, the determining unit 210 may also be used to: detect that the wrist node and head of the hand meet a first preset condition and that the hand belongs to the first hand shape; and / or detect that the palm node and head of the hand meet a second preset condition and that the hand belongs to the first hand shape.

[0231] In some embodiments, when the determining unit 210 detects that the wrist node of the hand and the head meet the first preset condition, it can be used to: calculate the first connecting line between the wrist node and the head; and detect that the angle between the wrist pose information of the wrist node and / or the head pose information of the head and the first connecting line meets the first preset condition.

[0232] In some embodiments, when determining whether the palm node of the hand and the head meet the second preset condition, the determining unit 210 may be used to: calculate the second connecting line between the palm node and the head, wherein the palm pose information of the palm node is determined at least based on the pose information of the first phalanx of one of the fingers of the hand and the wrist pose information of the wrist node of the hand; and detect that the angle between the palm pose information of the palm node and / or the head pose information of the head and the second connecting line meets the second preset condition.

[0233] In some embodiments, the determining unit 210 can also be used to: determine the pose of the second hand shape based on the wrist pose information of the wrist node of the hand.

[0234] In some embodiments, the duration of the second hand shape before release is less than a first time threshold.

[0235] In some embodiments, the interaction unit 220 can also be used to: cancel triggering the target function in response to the second hand shape maintaining for a period of time exceeding a first time threshold before release.

[0236] In some embodiments, the second hand shape being released at the target pose includes: the hand exiting the second hand shape at the target pose and switching to the release hand shape within a second time threshold.

[0237] In some embodiments, the interaction unit 220 can also be used to: cancel triggering the target function in response to the hand exiting the second hand shape and not switching to the release hand shape within the second time threshold.

[0238] In some embodiments, the interaction unit 220 can also be used to: display the function icon corresponding to the target function in response to the second hand being in the target pose.

[0239] In some embodiments, the interaction unit 220 can also be used to adjust the display of the function icons according to the direction of movement of the hand and / or the first relative distance between the head and the hand.

[0240] In some embodiments, when the interaction unit 220 adjusts the display of the function icon based on the movement direction of the hand and / or the first relative distance between the head and the hand, it can be used to: if the movement direction is horizontal, perform interpolation calculation based on a preset weighting factor, the initial display position of the function icon, and the current palm position of the hand to obtain a first interpolation result, and adjust the display position of the function icon based on the first interpolation result; or if the movement direction is vertical, and the first relative distance is greater than a second relative distance, perform interpolation calculation based on a preset weighting factor, the initial display position of the function icon, and the current palm position of the hand to obtain a second interpolation result, and adjust the display position of the function icon based on the second interpolation result, wherein the second relative distance is the distance between the initial display position and the head; or if the movement direction is vertical, and the first relative distance is less than the second relative distance, control the display position of the function icon to maintain a fixed distance from the head in the vertical direction; wherein the vertical direction is the direction of forward and backward movement along the connecting line between the head and the hand; the horizontal direction is the direction of left and right movement along a line perpendicular to the connecting line.

[0241] In some embodiments, the determining unit 210 may also be used to: determine the hand shape of the hand based on at least one of the values ​​of the flexion angle, bending angle and abduction angle of the finger joint of at least one finger of the hand.

[0242] In some embodiments, the determining unit 210 may also be used to: dynamically adjust the weight of at least one of the flexion angle, bending angle and abduction angle based on the wrist posture information of the hand.

[0243] In some embodiments, the interaction unit 220 can also be used to: determine a trigger area corresponding to the target function before triggering the target function, wherein the range of the trigger area is determined based on the initial palm pose when the hand is switched to the second hand shape; and prohibit triggering the target function when the hand is detected to be outside the trigger area.

[0244] In some embodiments, the interaction unit 220 can also be used to adjust the size of the trigger area according to the moving speed of the hand of the second hand away from the initial center point position of the trigger area.

[0245] In some embodiments, when the interaction unit 220 adjusts the size of the trigger area according to the moving speed of the second hand away from the initial center point position of the trigger area, it can be used to: when the moving speed of the second hand away from the initial center point position exceeds a speed threshold, the faster the moving speed, the smaller the trigger area; or when the moving speed of the second hand away from the initial center point position does not exceed the speed threshold, the trigger area maintains the initial area.

[0246] In some embodiments, the interaction unit 220 can also be used to display a visual indication of the trigger area.

[0247] In some embodiments, the interaction unit 220 can also be used to: hide the visual indication of the trigger area when the first relative distance between the head and the hand exceeds a first distance threshold.

[0248] In some embodiments, the interaction unit 220 can also be used to: cancel triggering the target function in response to the second hand being in the target pose and the descent distance exceeding the second distance threshold.

[0249] In some embodiments, the first target function is a homepage access function; or the second target function is a reset center function.

[0250] Each unit in the aforementioned interactive device 200 can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in the terminal device in hardware form, or stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0251] The interactive device 200 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the interactive device 200 can be the terminal or server.

[0252] In some embodiments, this disclosure also provides a terminal device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0253] As shown in Figure 13, which is a schematic diagram of the structure of a terminal device provided in an embodiment of this disclosure, the terminal device 300 can typically be provided in the form of glasses, a head-mounted display (HMD), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of the terminal device is not limited to these, and it can be further miniaturized or enlarged as needed. The terminal device 300 may include, but is not limited to, the following components:

[0254] Detection module 301: Uses various sensors to detect user operation commands and apply them to the virtual environment, such as continuously updating the images displayed on the screen according to the user's gaze, realizing the interaction between the user and the virtual environment and the scene, such as continuously updating the real content based on the detected direction of the user's head rotation.

[0255] Feedback module 302: Receives data from sensors and provides real-time feedback to the user; wherein, the feedback module 302 can be used to display a graphical user interface, such as displaying a virtual environment on the graphical user interface. For example, the feedback module 302 may include a display screen, etc.

[0256] Sensor 303: On the one hand, it receives operation commands from the user and applies them to the virtual environment; on the other hand, it provides the results of the operation to the user in various forms of feedback.

[0257] Control module 304: Controls sensors and various input / output devices, including acquiring user data (such as actions and voice) and outputting sensor data, such as images, vibrations, temperatures, and sounds, which affect the user, the virtual environment, and the real world.

[0258] Modeling module 305: Constructs a 3D model of the virtual environment, and may also include various feedback mechanisms such as sound and touch in the 3D model.

[0259] In this embodiment of the disclosure, a virtual scene in a three-dimensional environment can be constructed by the modeling module 305; the three-dimensional environment generated by the extended reality device can be displayed by the feedback module 302; the head and hand pose information of the target object can be detected by the detection module 301 and / or the sensor 303; the control module 304 determines that the hand is switched from the first hand shape to the second hand shape, and in response to the release of the second hand shape in the target pose, triggers the target function corresponding to the target pose; and the user interaction interface corresponding to the target function is displayed in the three-dimensional environment by the feedback module 302.

[0260] In some embodiments, as shown in FIG14, which is another structural schematic diagram of a terminal device provided in an embodiment of the present disclosure, the terminal device 300 further includes a processor 310 having one or more processing cores, a memory 320 having one or more computer-readable storage media, and a computer program stored on the memory 320 and executable on the processor. The processor 310 is electrically connected to the memory 320. Those skilled in the art will understand that the terminal device structure shown in the figures does not constitute a limitation on the terminal device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0261] The processor 310 is the control center of the terminal device 300. It connects various parts of the terminal device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 320, and calling data stored in the memory 320, it executes various functions of the terminal device 300 and processes data, thereby performing overall monitoring of the terminal device 300.

[0262] In this embodiment of the disclosure, the processor 310 in the terminal device 300 loads the instructions corresponding to the processes of one or more applications into the memory 320 according to the following steps, and the processor 310 runs the applications stored in the memory 320 to realize various functions:

[0263] The hand is determined to switch from the first hand shape to the second hand shape; in response to the release of the second hand shape in the target pose, the target function corresponding to the target pose is triggered.

[0264] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0265] In some embodiments, the processor 310 may include a detection module 301, a control module 304, and a modeling module 305.

[0266] In some embodiments, as shown in FIG14, the terminal device 300 further includes: a radio frequency circuit 306, an audio circuit 307, and a power supply 308. The processor 310 is electrically connected to the memory 320, the feedback module 302, the sensor 303, the radio frequency circuit 306, the audio circuit 307, and the power supply 308, respectively. Those skilled in the art will understand that the terminal device structure shown in FIG13 or FIG14 does not constitute a limitation on the terminal device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0267] The radio frequency circuit 306 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other terminal devices, and to transmit and receive signals with network devices or other terminal devices.

[0268] Audio circuit 307 can be used to provide an audio interface between a user and a terminal device via a speaker and a microphone. Audio circuit 307 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 307, converted back into audio data, and then processed by processor 310 before being transmitted via radio frequency circuit 306 to, for example, another terminal device, or output to memory for further processing. Audio circuit 307 may also include an earphone jack to provide communication between peripheral headphones and the terminal device.

[0269] Power supply 308 is used to supply power to the various components of terminal device 300.

[0270] Although not shown in Figure 13 or Figure 14, the terminal device 300 may also include a camera, a wireless fidelity module, a Bluetooth module, an input module, etc., which will not be described in detail here.

[0271] In some embodiments, this disclosure also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a terminal device or a server, and the computer program causes the terminal device or server to execute corresponding processes in the interaction methods of the embodiments of this disclosure; for brevity, further details are omitted here.

[0272] In some embodiments, this disclosure also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a terminal device reads the computer program from the computer-readable storage medium and executes the computer program, causing the terminal device to perform the corresponding flow in the interaction method of the embodiments of this disclosure; for brevity, further details are omitted here.

[0273] This disclosure also provides a computer program, which includes a computer program stored in a computer-readable storage medium. The processor of a terminal device reads the computer program from the computer-readable storage medium and executes the computer program, causing the terminal device to perform the corresponding flow in the interaction method of the embodiments of this disclosure; for brevity, further details are omitted here.

[0274] It should be understood that the processor in this disclosure embodiment may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0275] It is understood that the memory in the embodiments of this disclosure can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0276] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0277] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0278] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0279] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0280] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, the functional units in the embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0282] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to existing technology, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0283] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An interaction method, comprising: Determine whether to switch the hand from the first hand position to the second hand position; In response to the release of the second hand shape in the target pose, the target function corresponding to the target pose is triggered.

2. The interaction method according to claim 1, wherein, The step of responding to the release of the second hand shape in the target pose and triggering the target function corresponding to the target pose includes: If the second hand gesture is released in the first pose, the first target function corresponding to the first pose is triggered; or If the second hand gesture is released in the second pose, the second target function corresponding to the second pose is triggered.

3. The interaction method according to claim 1 or 2, further comprising: Display the user interface corresponding to the target function.

4. The interaction method according to any one of claims 1-3, wherein, Before determining the hand position from the first hand shape to the second hand shape, the method further includes: The detection method determines that the wrist joint of the hand and the head meet a first preset condition and the hand belongs to a first hand shape; and / or The detection method determines that the palm node of the hand and the head meet the second preset condition and the hand belongs to the first hand type.

5. The interaction method according to claim 4, wherein, The detection of whether the wrist node of the hand and the head meet the first preset condition includes: Calculate the first connection line between the wrist node and the head; The angle between the wrist pose information of the wrist node and / or the head pose information of the head and the first connecting line is determined to meet a first preset condition.

6. The interaction method according to claim 4 or 5, wherein, The step of detecting whether the palm node of the hand and the head meet the second preset condition includes: Calculate the second connection line between the palm node and the head. The palm pose information of the palm node is determined based at least on the pose information of the first phalanx of one of the fingers of the hand and the wrist pose information of the wrist node of the hand. The angle between the palm pose information of the detected palm node and / or the head pose information of the detected head and the second connecting line satisfies the second preset condition.

7. The interaction method according to any one of claims 1-6, further comprising: The pose of the second hand shape is determined based on the wrist pose information of the wrist node of the hand.

8. The interaction method according to any one of claims 1-7, wherein, The duration of the second hand shape before release is less than the first time threshold.

9. The interaction method according to claim 7, further comprising: In response to the second hand shape being maintained for a period of time exceeding a first time threshold before release, the triggering of the target function is cancelled.

10. The interaction method according to any one of claims 1-9, wherein, The second hand gesture is released at the target pose, including: The hand exits the second hand shape in the target pose and switches to the release hand shape within the second time threshold.

11. The interaction method according to claim 10, further comprising: In response to the hand exiting the second hand shape and not switching to the release hand shape within the second time threshold, the triggering of the target function is cancelled.

12. The interaction method according to any one of claims 1-11, further comprising: In response to the second hand being in the target pose, the function icon corresponding to the target function is displayed.

13. The interaction method according to claim 12, further comprising: The display of the function icon is adjusted according to the direction of movement of the hand and / or the first relative distance between the head and the hand.

14. The interaction method according to claim 13, wherein, Adjusting the display of the function icon based on the direction of hand movement and / or the first relative distance between the head and the hand includes: If the movement direction is horizontal, an interpolation calculation is performed based on a preset weighting factor, the initial display position of the function icon, and the current palm position of the hand to obtain a first interpolation result, and the display position of the function icon is adjusted according to the first interpolation result; or If the movement direction is vertical and the first relative distance is greater than the second relative distance, then an interpolation result is obtained by interpolating based on a preset weighting factor, the initial display position of the function icon, and the current palm position of the hand. The display position of the function icon is then adjusted according to the second interpolation result, wherein the second relative distance is the distance between the initial display position and the head; or If the direction of movement is vertical and the first relative distance is less than the second relative distance, then the display position of the function icon is controlled to maintain a fixed distance from the head in the vertical direction; Wherein, the longitudinal direction is the direction of movement back and forth along the line connecting the head and the hand; the lateral direction is the direction of movement left and right along the line perpendicular to the connecting line.

15. The interaction method according to any one of claims 1-14, further comprising: The hand shape of the hand is determined based on at least one of the values ​​of flexion angle, bending angle, and abduction angle of the finger joint of at least one finger of the hand.

16. The interaction method according to claim 15, further comprising: Based on the wrist position information of the hand, the weight of at least one of the values ​​of flexion angle, bending angle and abduction angle is dynamically adjusted.

17. The interaction method according to any one of claims 1-16, further comprising: Before triggering the target function, the trigger area corresponding to the target function is determined, and the range of the trigger area is determined according to the initial palm position when the hand is switched to the second hand shape; When the hand is detected to be outside the trigger area, the target function is disabled.

18. The interaction method according to claim 17, further comprising: The size of the trigger area is adjusted according to the moving speed of the second hand away from the initial center point position of the trigger area.

19. The interaction method according to claim 18, wherein, Adjusting the size of the trigger area based on the moving speed of the second hand away from the initial center point position of the trigger area includes: When the movement speed of the second hand away from the initial center point exceeds a speed threshold, the faster the movement speed, the smaller the trigger area; or When the movement speed of the second hand away from the initial center point does not exceed the speed threshold, the trigger area maintains the initial area.

20. The interaction method according to any one of claims 17-19, further comprising: A visual indication of the trigger area is displayed.

21. The interaction method according to claim 20, wherein, When the first relative distance between the head and the hand exceeds a first distance threshold, the visual indication of the trigger area is hidden.

22. The interaction method according to any one of claims 1-21, further comprising: In response to the second hand being in the target pose and the descent distance exceeding the second distance threshold, the triggering of the target function is cancelled.

23. The interaction method according to claim 2, wherein, The first target function is the homepage access function; or the second target function is the reset center function.

24. An interactive device, comprising: The determining unit is configured to determine when the hand changes from a first hand shape to a second hand shape; The interaction unit is configured to trigger the target function corresponding to the target pose in response to the release of the second hand shape in the target pose.

25. A computer-readable storage medium storing a computer program, wherein, The computer program is adapted to be loaded by a processor to perform the interactive method as described in any one of claims 1-23.

26. A terminal device, comprising a processor and a memory, wherein, The memory stores a computer program, and the processor executes the interactive method according to any one of claims 1-23 by calling the computer program stored in the memory.

27. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the interaction method according to any one of claims 1-23.

Images