Fingertip interaction method and device, head-mounted equipment and storage medium

Abstract

The invention relates to the technical field of human-computer interaction, and particularly discloses a fingertip interaction method and device, head-mounted equipment and a storage medium. According to the method, the three-dimensional coordinates of the fingertips of the user can be recognized, the gesture of the user and / or the movement track of the fingertips of the user on the space coordinate system can be recognized according to changes of the three-dimensional coordinates of the fingertips, and then the target interaction event is determined according to the gesture of the user and / or the movement track. The fingertip can freely move in a three-dimensional space to perform complex operation, interaction on a plane can be performed, interaction on a depth coordinate axis can also be performed, and the interaction flexibility is improved; besides, according to the change of the three-dimensional coordinates of the fingertip, the movement track of the fingertip of the user can be accurately identified, so that the target interaction event is accurately matched, and the interaction accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a fingertip interaction method, device, head-mounted device and storage medium. Background Art

[0002] With the development of computer technology, more and more augmented reality or virtual reality smart wearable devices (such as AR glasses, smart helmets, etc.) are beginning to enter the market, but the interaction methods of most products still have some limitations, especially in terms of the accuracy, naturalness and flexibility of gesture control in space. Currently, the commonly used interaction methods include ring control, touching the temples, head shaking, eye tracking and voice control. Among them, although the ring control can sense the movement of fingers through the ring, it relies on external hardware, has a heavy burden to wear, and has limited operating space and cannot be freely controlled, resulting in inaccurate operation results; the interaction method of touching the temples has limited operating space and cannot effectively support precise control in three-dimensional space; the interaction method of head shaking control has low accuracy, and long-term use is easy to cause fatigue, and it is difficult to combine head movement with gestures, making it difficult to provide accurate three-dimensional space operations; the interaction methods of eye tracking and voice control cannot provide sufficient flexibility and accuracy when controlling complex three-dimensional spaces.

[0003] In summary, the currently commonly used interaction methods cannot achieve flexible and accurate interaction in three-dimensional space. Therefore, how to improve the flexibility and accuracy of interaction has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a fingertip interaction method, apparatus, head-mounted device and storage medium to improve the flexibility and accuracy of interaction.

[0005] In a first aspect, the present application provides a fingertip interaction method, the method comprising:

[0006] Based on a camera corresponding to the head mounted device, a hand image set of the user is collected, and each hand image in the hand image set is detected to obtain a fingertip two-dimensional coordinate set corresponding to the fingertip of the user;

[0007] Acquire camera parameters of the camera, and calculate a three-dimensional coordinate set of a fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip;

[0008] Based on the fingertip three-dimensional coordinate set, the user's gesture action and / or the movement trajectory of the user's fingertip in the spatial coordinate system are identified, and according to the gesture action and / or the movement trajectory, a target interaction event is determined to interact based on the interaction event.

[0009] In a second aspect, the present application further provides a fingertip interaction device, the device comprising:

[0010] A two-dimensional coordinate recognition module, used to collect a hand image set of the user based on a camera corresponding to the head mounted device, and detect each hand image in the hand image set to obtain a fingertip two-dimensional coordinate set corresponding to the user's fingertips;

[0011] A three-dimensional coordinate calculation module, used for obtaining camera parameters of the camera, and calculating a three-dimensional coordinate set of the fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip;

[0012] An interaction event determination module is used to identify the user's gesture actions and / or the movement trajectory of the user's fingertips in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determine the target interaction event according to the gesture actions and / or the movement trajectory, so as to interact based on the interaction event.

[0013] In a third aspect, the present application also provides a head-mounted device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the fingertip interaction method as described above when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the fingertip interaction method as described above.

[0015] The present application discloses a fingertip interaction method, device, head-mounted device and storage medium. Based on a camera corresponding to the head-mounted device, a user's hand image set is collected, and each hand image in the hand image set is detected to obtain a fingertip two-dimensional coordinate set corresponding to the user's fingertips; the camera parameters of the camera are obtained, and a fingertip three-dimensional coordinate set is calculated based on the camera parameters and the fingertip two-dimensional coordinate set; based on the fingertip three-dimensional coordinate set, the user's gesture action and / or the movement trajectory of the user's fingertip in the spatial coordinate system are identified, and according to the gesture action and / or the movement trajectory, a target interaction event is determined to interact based on the interaction event. The present application can identify the user's gesture and / or the movement trajectory of the user's fingertip in the spatial coordinate system by identifying the three-dimensional coordinates of the user's fingertip, and according to the changes in the three-dimensional coordinates of the fingertip, the user's gesture and / or the movement trajectory of the user's fingertip in the spatial coordinate system can be identified, and then the target interaction event can be determined according to the user's gesture and / or the movement trajectory. Fingertips can move freely in three-dimensional space to perform complex operations. They can interact not only on the plane but also on the depth coordinate axis, which improves the flexibility of the interaction. In addition, based on the changes in the three-dimensional coordinates of the fingertips, the movement trajectory of the user's fingertips can be accurately identified, and then the target interaction events can be accurately matched, improving the accuracy of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a schematic flow chart of a fingertip interaction method provided in the first embodiment of the present application;

[0018] Figure 2 is a schematic diagram of key points of a hand provided in an embodiment of the present application;

[0019] Figure 3 is a schematic flow chart of a fingertip interaction method provided in the second embodiment of the present application;

[0020] Figure 4 A schematic block diagram of a fingertip interaction device provided in an embodiment of the present application;

[0021] Figure 5 A schematic block diagram of the structure of a head-mounted device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] The embodiments of the present application provide a fingertip interaction method, device, head-mounted device and storage medium. Among them, the fingertip interaction method can be applied to a head-mounted device, by identifying the three-dimensional coordinates of the user's fingertips, and tracking the movement trajectory of the user's fingertips in the spatial coordinate system according to the changes in the three-dimensional coordinates of the fingertips, thereby determining the target interaction event. The fingertips can move freely in three-dimensional space to perform complex operations, and can not only interact on the plane but also on the depth coordinate axis, thereby improving the flexibility of the interaction; in addition, according to the changes in the three-dimensional coordinates of the fingertips, the movement trajectory of the user's fingertips can be accurately identified, thereby accurately matching the target interaction event, thereby improving the accuracy of the interaction.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] See also Figure 1 , Figure 1 FIG. 1 is a schematic flow chart of a fingertip interaction method provided by the first embodiment of the present application. Figure 1 As shown, the fingertip interaction method specifically includes steps S101 to S103.

[0029] S101, based on a camera corresponding to the head mounted device, collecting a hand image set of the user, and detecting each hand image in the hand image set to obtain a fingertip two-dimensional coordinate set corresponding to the fingertip of the user;

[0030] In one embodiment, the head-mounted device may be an augmented reality device, a virtual reality device, a mixed display device, etc. (such as AR glasses, smart helmets, etc.). When the user wears the head-mounted device, a camera configured in the head-mounted device or connected thereto is used to collect the user's hand image set or hand video stream, and a hand key point detection algorithm is used to identify the fingertip two-dimensional coordinate set corresponding to the user's fingertip. The collection of the hand image set may be performed continuously according to a preset interval time.

[0031] Among them, the hand key point detection model is a technology used in the field of computer vision to identify and locate the key positions of the hand (such as knuckles and wrists) in images or videos. For example, the PP-TinyPose model, the YOLOv8 model, the HRNet and LiteHRNet models, etc.

[0032] In one embodiment, the collected hand images and hand video streams are preprocessed, specifically, the hand images or video frames in the hand video streams are cropped, rotated, scaled, and binarized to improve the recognition accuracy of the key points of the hand.

[0033] The hand key point detection model detects the preprocessed hand image or the video frame of the hand video stream, first identifies the hand outline, and then extracts 21 hand key points including the user's fingertips, such as Figure 2 As shown. And extract the key point ID of the fingertip from the 21 key points of the hand. The key point ID of the fingertip is usually predefined, for example, Figure 2 Tip I ds = [4, 8, 12, 16, 20] in the figure can represent the IDs of the five fingertips of thumb, index finger, middle finger, ring finger and little finger respectively. Finally, the two-dimensional coordinates (x, y) of each key point can be obtained by parsing the heatmap output by the hand key point detection model.

[0034] Among them, heatmap is a data visualization technology that uses blocks of different colors to represent the distribution density of data, which can intuitively understand the distribution pattern of data. In the key point detection task, heatmap performs feature matching in the spatial dimension and obtains the position of the key point by finding the index of the maximum response point, that is, the two-dimensional coordinate (x, y). The two-dimensional coordinate system usually takes the upper left corner of the hand image as the origin (0, 0), where the x-axis represents the horizontal direction of the image and the y-axis represents the vertical direction of the image.

[0035] In one embodiment, the user's fingertips and the fingertip two-dimensional coordinate set identified by the hand key point detection model are used. For example, the coordinates of the index fingertip can be obtained by obtaining the coordinates of the key point ID 8. The coordinates of other fingertips are also obtained in the same way.

[0036] In another embodiment, before performing fingertip two-dimensional coordinate recognition, the hand image or hand video stream captured by the camera can also be recognized to preliminarily determine whether the user's purpose is interaction. Specifically, the current gesture obtained by recognition can be compared with the preset interaction trigger gesture. When the current gesture is the same as the interaction trigger gesture, it is determined that the user is performing fingertip interaction.

[0037] S102, obtaining camera parameters of the camera, and calculating a three-dimensional coordinate set of a fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip;

[0038] In one embodiment, the camera parameters include the focal length of a camera configured in or correspondingly connected to a head mounted device worn by the user.

[0039] According to the preset depth coordinate value calculation formula, camera parameters and the two-dimensional coordinates of the fingertip, the depth coordinate value of the fingertip is calculated. Then, according to the three-dimensional coordinate calculation formula, the depth coordinate value of the fingertip and the two-dimensional coordinates of the fingertip, the three-dimensional coordinates of the fingertip are calculated to generate a three-dimensional coordinate set of the fingertip.

[0040] In one embodiment, based on the camera parameters and the fingertip two-dimensional coordinate set, a fingertip three-dimensional coordinate set is calculated, including: based on a preset depth coordinate value calculation formula, the fingertip two-dimensional coordinates and the camera parameters, calculating the depth coordinate value of the user's fingertip in the spatial coordinate system; based on a preset three-dimensional coordinate calculation formula and the depth coordinate value, calculating the fingertip three-dimensional coordinates to generate the fingertip three-dimensional coordinate set.

[0041] The depth coordinate value of each fingertip is calculated using the depth coordinate value calculation formula and the two-dimensional coordinates of each fingertip. The depth coordinate value calculation formula is:

[0042]

[0043] Where f is the focal length of the camera, d is the actual distance between the fingertip and the camera, x is the horizontal coordinate value in the two-dimensional coordinates of the fingertip, Z is the depth coordinate value in the three-dimensional coordinates of the fingertip, and * represents the product.

[0044] After obtaining the depth coordinate value, the three-dimensional coordinates (X, Y, Z) of the fingertip are calculated using the three-dimensional coordinate calculation formula. The three-dimensional coordinate calculation formula is:

[0045] All the fingertip 3D coordinates in the hand image set or the hand video stream and the fingertip 3D coordinates in the previous and next frame images are integrated to generate a fingertip 3D coordinate set.

[0046] It can be understood that the fingertip two-dimensional coordinate set includes the fingertip IDs that appear in the hand image set or the hand video stream and the two-dimensional coordinates of each fingertip in each frame image. The hand image set contains all images of the user's hand during movement, and the number of coordinates in the fingertip two-dimensional coordinate set depends on the number of fingertips that appear in the hand image set and the number of images collected during the hand movement. The fingertip three-dimensional coordinates are calculated from the fingertip two-dimensional coordinates, so the number of coordinates in the fingertip three-dimensional coordinate set is the same as the number in the fingertip two-dimensional coordinate set.

[0047] S103. Based on the fingertip three-dimensional coordinate set, identify the user's gesture action and / or the movement trajectory of the user's fingertip in the spatial coordinate system, and determine the target interaction event according to the gesture action and / or the movement trajectory, so as to interact based on the interaction event.

[0048] In one embodiment, a target tracking algorithm (such as Kalman filtering or optical flow method, etc.) is used to continuously track the motion trajectory of the fingertips during the user's hand movement. Specifically, the three-dimensional coordinates of the fingertips in each frame of the hand image set or the hand video stream are obtained and compared to obtain the change information of the three-dimensional coordinates of the fingertips. According to the change of the three-dimensional coordinates of the fingertips, the motion trajectory of the user's fingertips is identified, and then according to the user's gesture and movement state, the target interaction event is determined to achieve interaction.

[0049] Exemplarily, the Kalman filter algorithm is used to track the fingertips and identify the fingertip motion trajectory. The formula for updating the three-dimensional coordinates of the fingertips is as follows:

[0050]

[0051] in, is the estimated value of the three-dimensional coordinates of the fingertip at the kth moment based on the measured value of the three-dimensional coordinates of the fingertip at the kth moment, and K is the estimated value of the three-dimensional coordinates of the fingertip at the kth moment based on the measured value of the three-dimensional coordinates of the fingertip at the k-1th moment. k is the Kalman gain, z k is the measured value of the three-dimensional coordinates of the fingertip at time k, H k is the observation matrix, which is used to map the estimated values ​​to the observation space, that is, to convert the estimated values ​​into a form that can be compared with the actual values.

[0052] It is understandable that the user's hand movement can occur in three directions of the spatial coordinate system, and the fingertip movement trajectory in each direction of the spatial coordinate system can correspond to different interaction events. Different interaction events can be determined according to the different movement trajectories of the user's fingertips in the spatial coordinate system as target interaction events for interaction.

[0053] It is understandable that the triggering condition of the interaction event can be a separate gesture action or a fingertip movement under the condition of a gesture action. For example, when the user performs a pinch operation, the triggered interaction event is waking up the main menu; when the user performs a pinch operation and moves on the depth coordinate axis at the same time, the triggered interaction event may be switching the main menu.

[0054] Furthermore, based on the fingertip three-dimensional coordinate set, the movement trajectory of the user's fingertips in the spatial coordinate system is identified, and according to the movement trajectory, a target interaction event is determined, including: based on the fingertip three-dimensional coordinate set, obtaining change information of the fingertip three-dimensional coordinates, and based on the change information, identifying the movement trajectory of the user's fingertips; obtaining the interaction event corresponding to the movement trajectory from the interactive event list as the target interaction event.

[0055] In one embodiment, for each fingertip, the change information of the fingertip in the continuous frame images is calculated according to the three-dimensional coordinates of the fingertip in the three-dimensional coordinate set of the fingertip, and the movement trajectory of the fingertip is identified according to the change information.

[0056] The interactive event list may include interactive elements, interactive events, and fingertip motion trajectories that trigger interactive events. The identified fingertip motion trajectory is matched with the fingertip motion trajectories in the interactive event list to determine the interactive event corresponding to the identified fingertip motion trajectory, and the interactive event is used as the target interactive event.

[0057] Furthermore, based on the fingertip three-dimensional coordinate set, the user's gesture action is identified, and according to the gesture action, the target interaction event is determined, including: based on the fingertip three-dimensional coordinate set, obtaining the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip; based on the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip, calculating the fingertip distance between the first target fingertip and the second target fingertip; when the fingertip distance is less than a second preset distance threshold, determining that the gesture action is a fingertip pinch, and obtaining the interaction event corresponding to the fingertip pinch as the target interaction event.

[0058] In one embodiment, the position change of the fingertip can be identified based on the fingertip three-dimensional coordinate set, and the user's gesture action (such as fingertip pinching) or fingertip movement trajectory can be accurately identified based on the fingertip position change.

[0059] Exemplarily, the change in the distance between the first target fingertip and the second target fingertip is detected through the three-dimensional coordinates of the fingertips of the first target fingertip and the second target fingertip to determine whether the user has performed a pinch operation. For example, when the distance between the thumb (the first target fingertip) and the index finger (the second target fingertip) is less than the second preset distance threshold, it is considered that the user has performed a pinch operation, and an interaction event corresponding to the pinch operation (for example, waking up the main menu) is obtained, and the interaction event is executed to wake up the main menu for the user to operate.

[0060] The calculation formula of fingertip distance is:

[0061]

[0062] x thumb ,y thumb 、z thumb is the three-dimensional coordinate value of the thumb, x index ,y index 、z index is the three-dimensional coordinate value of the index finger.

[0063] Furthermore, the spatial coordinate system includes a depth coordinate axis, a horizontal coordinate axis and a vertical coordinate axis, and the method of identifying the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determining the target interaction event based on the motion trajectory, also includes: based on the fingertip three-dimensional coordinate set, identifying the change information of the user's fingertip in the spatial coordinate system, obtaining the first motion trajectory of the user's fingertip in the direction of the depth coordinate axis, the second motion trajectory of the user's fingertip on the vertical coordinate axis, and the third motion trajectory of the user's fingertip on the horizontal coordinate axis; obtaining the interaction event corresponding to at least one of the first motion trajectory, the second motion trajectory and the third motion trajectory from the interactive event list as the target interaction event.

[0064] In one embodiment, the movement trajectory of the user's fingertip on the depth coordinate axis (Z axis), the ordinate axis (Y axis) and the abscissa axis (X axis) is identified based on the change in the three-dimensional coordinate of the fingertip. Specifically, the movement data of the fingertip on the Z axis reflects the forward and backward movement of the fingertip in space. The movement data of the fingertip on the Y axis reflects the movement of the fingertip in the vertical direction. The movement data of the fingertip on the X axis reflects the movement of the fingertip in the horizontal direction.

[0065] In one embodiment, based on at least one of the first motion trajectory, the second motion trajectory and the third motion trajectory of the user's fingertip on the Z axis, the Y axis and the X axis, a comparison is made in the list of interactive events to find the best matching motion trajectory, and the corresponding interactive event is extracted as the target interactive event.

[0066] Exemplarily, when the movement of the fingertip along the Z axis (front-back direction) is detected, the first motion track is identified, and the interactive event corresponding to the first motion track is matched from the interactive event list as the target interactive event. For example, when the first motion track is a push along the Z axis, the interactive event corresponding to the current fingertip track movement is identified from the interactive event list as "enter the next level menu". When the first motion track is a pull back along the Z axis, the interactive event corresponding to the current fingertip track movement is identified from the interactive event list as "return to the previous level menu".

[0067] Exemplarily, the motion trajectory of the user's fingertip in the X-axis (left-right direction) and the Y-axis (up-down direction) can be used to switch between menu items or different applications. When the third motion trajectory is sliding the fingertip to the right along the X-axis, it will switch to the next menu item or application. When the third motion trajectory is sliding the fingertip to the left along the X-axis, it will switch to the previous menu item or application. When the second motion trajectory is sliding the fingertip up and down along the Y-axis, the menu items are switched vertically for navigation in long lists.

[0068] Further, obtaining an interactive event corresponding to at least one of the first motion trajectory, the second motion trajectory and the third motion trajectory from the interactive event list as the target interactive event includes: based on at least one of the first motion trajectory, the second motion trajectory and the third motion trajectory, identifying movement parameters of the user's fingertip in the spatial coordinate system, the movement parameters including at least one of a moving speed, a rotation angle and a moving distance; and obtaining an interactive event corresponding to the movement parameters from the interactive event list as the target interactive event.

[0069] In one embodiment, the movement parameters of the user's fingertip on each axis of the spatial coordinate system are identified based on the movement trajectory of the fingertip on each axis, that is, the movement speed, rotation angle and movement distance of the user's fingertip on each axis of the spatial coordinate system are identified. It can be understood that when the fingertip moves only in the direction of the depth coordinate axis, there is only a first movement trajectory; when the fingertip moves only on the ordinate axis, there is only a second movement trajectory; when the fingertip moves only on the abscissa axis, there is only a third movement trajectory. When the fingertip has a movement trajectory only on a single coordinate axis, it is only necessary to calculate the movement speed of the fingertip on that axis.

[0070] When the first motion trajectory, the second motion trajectory and the third motion trajectory all exist, the moving speed of the fingertip can be determined by calculating the change in the three-dimensional coordinate position of the fingertip in consecutive frames and dividing it by the time interval. The moving speed calculation formula is:

[0071]

[0072] Among them, (x k ,y k , z k ) is the three-dimensional coordinate of the fingertip in the current frame image, (x k-1 ,y k-1 , z k-1 ) is the 3D coordinate of the fingertip in the previous frame, and Δt is the time interval between the two frames.

[0073] The rotation angle of the fingertip can be determined by calculating the change in angle of the fingertip relative to a reference point. Specifically, this is achieved by calculating the change in angle between two vectors, such as using the dot product and cross product of the vectors. The displacement of the fingertip can be determined by calculating the total displacement of the fingertip over a period of time to determine the moving distance.

[0074] Exemplarily, when the fingertip moves a certain distance along the Z axis, the selection operation is completed. For example, when the fingertip moves forward along the Z axis to a threshold, it is recognized as a "confirmation" operation. When the fingertip is in a specified area and stays for a certain period of time, it is recognized as a "selection" operation, triggering the activation of the current option. That is, the confirmation operation is completed by pushing the fingertip forward along the Z axis, and the user stays on a menu item, which is recognized as "selection" and activates the menu item.

[0075] For example, in addition to translation, you can also perform certain special operations by rotating your fingers (i.e., rotating around the X, Y, and Z axes), such as rotating virtual objects, switching different perspectives, etc. When the user rotates his fingers by a certain angle (around the Y axis), it will trigger an interactive event that rotates the virtual object or scene perspective, providing a richer way of interaction. Rotating the fingertips around the Z axis will trigger certain special effects of the interface, such as rotating menus, items, and perspectives. Specifically, when the user selects a virtual object or scene in the field of view and detects the rotation of the fingertips, the object will be rotated or the perspective will be adjusted accordingly, providing a more natural interactive experience.

[0076] Exemplarily, fast sliding or smooth scrolling can be achieved by fast movement of the fingertips (especially in the X and Y axis directions), which is suitable for scenarios such as long lists, image browsing, or quick browsing of menus. When the fingertip moves quickly in the X or Y axis direction, it is recognized as a fast scrolling operation to quickly browse the menu or application list. When the fingertip moves slowly in the X or Y axis direction, it is recognized as smooth scrolling, which is suitable for precise control, such as scrolling to view detailed content. That is, if the user's fingertip slides quickly on the menu list (the fingertip moves quickly), the list will scroll quickly, and if it slides slowly in the same menu (the fingertip moves slowly), the list will scroll smoothly.

[0077] It is understandable that the moving speed, moving distance and rotation angle of the fingertip in the spatial coordinate system can trigger an interaction event individually or in combination, and can be set by the user according to actual needs.

[0078] Further, obtaining the interaction events corresponding to the moving speed, the rotation angle and the moving distance from the interactive event list as the target interaction events includes: comparing a preset speed threshold, a preset angle threshold and a first preset distance threshold with the moving speed, the rotation angle and the moving distance respectively to obtain a comparison result; according to the comparison result, deleting the moving speed, the rotation angle and the moving distance that meet a preset exclusion condition to obtain a moving target speed, a rotation target angle and a moving target distance, wherein the preset exclusion condition includes a moving speed less than the speed threshold, a rotation angle less than the preset angle threshold and a moving distance less than the first preset distance threshold; obtaining the interaction events corresponding to the moving target speed, the rotation target angle and the moving target distance from the interactive event list as the target interaction event.

[0079] It is understandable that during the user's hand movement, there will be slight shaking, and there will be some movement distances, movement speeds, and rotation angles that are not intended by the user. For example, when the user only needs to move on the Y-axis, the hand shaking will cause movement in the X-axis or Z-axis direction in the recognition result. It is necessary to compare the moving distance with the first preset distance threshold to remove the invalid movement in the X-axis or Z-axis direction to avoid invalid movement in the X-axis or Z-axis direction affecting the matching results of the target interaction event. Therefore, in order to remove errors caused by slight hand shaking, the moving distance, moving speed or rotation angle that does not meet the requirements is deleted.

[0080] In one embodiment, the preset speed threshold, the preset angle threshold and the first preset distance threshold can be determined by the user according to specific application scenarios and interaction requirements, and are used to distinguish between valid gesture actions and motion trajectories and slight movements caused by hand shaking.

[0081] Specifically, the moving speed, rotation angle and moving distance of the fingertips are compared with the preset thresholds to determine which gestures are valid and which are slight movements caused by hand shaking. If the moving speed of the fingertips is less than the speed threshold, it is considered that the movement is caused by hand shaking and should be excluded. If the rotation angle of the fingertips is less than the angle threshold, it is considered that the rotation is caused by hand shaking and should be excluded. If the moving distance of the fingertips is less than the distance threshold, it is considered that the movement is caused by hand shaking and should be excluded.

[0082] In one embodiment, according to the above comparison results, the moving speed, rotation angle and moving distance that meet the preset exclusion conditions are deleted. The remaining moving speed, rotation angle and moving distance are considered valid and can be used as the moving target speed, rotation target angle and moving target distance for subsequent interaction event matching. It can effectively identify and exclude slight movements caused by hand shaking, and only respond to valid gesture actions, thereby improving the accuracy and stability of gesture interaction.

[0083] In one embodiment, an interactive event corresponding to the moving target speed, the rotating target angle, and the moving target distance is obtained from the interactive event list as a target interactive event. According to the target interactive event, a corresponding operation or response is performed.

[0084] It is understandable that after deleting the moving speed, rotation angle, and moving distance that meet the preset exclusion conditions, only the moving target speed, rotation target angle, or moving target distance may remain, or the moving target speed and rotation target angle, the moving target speed and moving target distance, or the rotation target angle and moving target distance may remain. When matching the target interaction event, matching is performed based on the information obtained after exclusion.

[0085] The above-mentioned embodiments provide a fingertip interaction method, apparatus, head-mounted device and storage medium, which collects a user's hand image set based on a camera corresponding to the head-mounted device, and detects each hand image in the hand image set to obtain a fingertip two-dimensional coordinate set corresponding to the user's fingertips; obtains the camera parameters of the camera, and calculates the fingertip three-dimensional coordinate set based on the camera parameters and the fingertip two-dimensional coordinate set; based on the fingertip three-dimensional coordinate set, identifies the user's gesture action and / or the movement trajectory of the user's fingertips in the spatial coordinate system, and determines the target interaction event based on the gesture action and / or the movement trajectory, so as to interact based on the interaction event. The present application can identify the user's gesture and / or the movement trajectory of the user's fingertips in the spatial coordinate system by identifying the three-dimensional coordinates of the user's fingertips, and according to the changes in the three-dimensional coordinates of the fingertips, and then determine the target interaction event based on the user's gesture and / or the movement trajectory. Fingertips can move freely in three-dimensional space to perform complex operations. They can interact not only on the plane but also on the depth coordinate axis, which improves the flexibility of the interaction. In addition, based on the changes in the three-dimensional coordinates of the fingertips, the movement trajectory of the user's fingertips can be accurately identified, and then the target interaction events can be accurately matched, improving the accuracy of the interaction.

[0086] See also Figure 2 , Figure 2This is a schematic flow chart of a fingertip interaction method provided in the second embodiment of the present application. The fingertip interaction method can be applied to a head-mounted device, can effectively extract interactive events from the display content of the user interface, and generate interactive prompt information to assist the user to interact with the interface through fingertip movements. It improves the naturalness and intuitiveness of user interaction, can effectively guide users to interact, and improves the portability and efficiency of interaction.

[0087] like Figure 2 As shown, the fingertip interaction method specifically includes steps S201 to S202.

[0088] S201, obtaining display content of a display interface;

[0089] S202. Based on the displayed content, extract the interactive events corresponding to the displayed content and the fingertip motion trajectories corresponding to the interactive events from a preset interactive event database, and generate an interactive event list for matching the target interactive events.

[0090] In one embodiment, the display content of the display interface is obtained. Specifically, it can be achieved by using a screen capture technology or directly accessing a rendering layer of the user interface.

[0091] It is understandable that when a user wears an augmented reality device (such as AR glasses), the display interface contains real objects. If interaction with the real objects is required, it is necessary to use the camera configured in the augmented reality device or corresponding to it to capture the image of the displayed object and convert the displayed object into an interactive element existing in the real interface.

[0092] In one embodiment, each interactive element in the displayed content is identified based on the acquired display content, and the interactive event corresponding to each interactive element is extracted from a preset interactive event database. The interactive event database includes interactive elements, interactive events corresponding to the interactive elements (such as rotation, movement, zooming in, zooming out, etc.), and fingertip motion trajectories corresponding to the interactive events (such as fingertip pinching, fingertip rotation around the horizontal axis, fingertip forward movement on the depth axis, etc.).

[0093] Specifically, image recognition technology may be used to identify each element in the displayed content, and match it with the interactive elements in the interactive event database to determine the interactive elements in the current displayed content.

[0094] In one embodiment, the extracted interactive events and the fingertip motion trajectories corresponding to the interactive events are integrated to generate an interactive event list for matching the target interactive events.

[0095] In another embodiment, corresponding interactive prompt information may also be generated according to the interactive event list. Specifically, the interactive event list includes interactive elements, interactive events, and trigger conditions corresponding to the interactive events. A preset prompt template is obtained, and according to the prompt template, the interactive elements, interactive events, and trigger conditions corresponding to the interactive events are processed to obtain the interactive prompt information.

[0096] The prompt template may be in the form of text, voice, etc. For example, when the prompt template is in text form, the interactive elements, interactive events, and trigger conditions corresponding to the interactive events may be filled into the variables of the text prompt template to generate interactive prompt information in text form, such as "swipe up with your fingertip to browse the menu list". When the prompt template is in voice form, the voice prompt information may be played.

[0097] In one embodiment, the interaction prompt information is intended to assist the user in understanding how to interact with the interface through fingertip motion. The interaction prompt information may also display the interactive events and interaction methods corresponding to each interactive element in the display interface. For example, if there is a scrollable list on the display interface, the interaction prompt information may remind the user that the current list is scrollable and the corresponding fingertip motion trajectory is fast or slow movement on the X-axis or Y-axis.

[0098] In one embodiment, the user performs fingertip movements according to the prompt information, and the user's fingertip movements are captured by gesture recognition technology and converted into corresponding interaction commands to achieve interaction between the user and the interface.

[0099] In the above embodiment, interactive events can be effectively extracted from the display content of the user interface, and interactive prompt information can be generated to assist the user to interact with the interface through fingertip movements. The naturalness and intuitiveness of user interaction are improved, and the user can be effectively guided to interact, thereby improving the portability and efficiency of interaction.

[0100] See also Figure 3 , Figure 3 The embodiment of the present application provides a schematic block diagram of a fingertip interaction device, which is used to execute the aforementioned fingertip interaction method. The fingertip interaction device can be configured in a head-mounted device.

[0101] like Figure 3 As shown, the fingertip interaction device 300 includes:

[0102] The two-dimensional coordinate recognition module 301 is used to collect a set of hand images of the user based on a camera corresponding to the head mounted device, and detect each hand image in the hand image set to obtain a two-dimensional coordinate set of fingertips corresponding to the fingertips of the user;

[0103] A three-dimensional coordinate calculation module 302, used to obtain camera parameters of the camera, and calculate a three-dimensional coordinate set of the fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip;

[0104] The interaction event determination module 303 is used to identify the user's gesture action and / or the movement trajectory of the user's fingertips in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determine the target interaction event according to the gesture action and / or the movement trajectory, so as to interact based on the interaction event.

[0105] Furthermore, the fingertip interaction device 300 further includes an interaction information prompting module, and the interaction information prompting module includes:

[0106] An interactive event list generating unit, used to obtain display content of the display interface;

[0107] Based on the displayed content, the interactive events corresponding to the displayed content and the fingertip motion trajectories corresponding to the interactive events are extracted from a preset interactive event database to generate an interactive event list for matching the target interactive events.

[0108] Furthermore, the interaction event determination module 303 includes:

[0109] A gesture recognition unit, configured to obtain change information of the three-dimensional coordinates of the fingertips based on the three-dimensional coordinate set of the fingertips, and to recognize a movement trajectory of the user's fingertips based on the change information;

[0110] The target interaction event determination unit is used to obtain the interaction event corresponding to the motion trajectory from the interactive event list as the target interaction event.

[0111] Furthermore, the spatial coordinate system includes a depth coordinate axis, a horizontal coordinate axis, and a vertical coordinate axis. The interaction event determination module 303 further includes:

[0112] a motion trajectory recognition unit, configured to recognize change information of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and obtain a first motion trajectory of the user's fingertip in the direction of the depth coordinate axis, a second motion trajectory of the user's fingertip in the longitudinal coordinate axis, and a third motion trajectory of the user's fingertip in the transverse coordinate axis;

[0113] The target interaction event determining unit is configured to obtain, from the interactive event list, an interaction event corresponding to at least one of the first motion trajectory, the second motion trajectory, and the third motion trajectory as the target interaction event.

[0114] Furthermore, the interaction event determination module 303 further includes:

[0115] a movement trajectory recognition module, configured to recognize movement parameters of the user's fingertip in the spatial coordinate system based on at least one of the first movement trajectory, the second movement trajectory, and the third movement trajectory, wherein the movement parameters include at least one of a movement speed, a rotation angle, and a movement distance;

[0116] The target interaction event determination unit is used to obtain the interaction event corresponding to the movement parameter from the interactive event list as the target interaction event.

[0117] Furthermore, the target interaction event determination unit includes:

[0118] a comparison result obtaining subunit, used to compare a preset speed threshold, a preset angle threshold and a first preset distance threshold with the moving speed, the rotation angle and the moving distance respectively, to obtain a comparison result;

[0119] a motion trajectory deletion subunit, configured to delete the moving speed, rotation angle, and moving distance that meet the preset exclusion condition according to the comparison result, and obtain the moving target speed, the rotating target angle, and the moving target distance, wherein the preset exclusion condition includes that the moving speed is less than the speed threshold, the rotation angle is less than the preset angle threshold, and the moving distance is less than the first preset distance threshold;

[0120] The target interaction event determination subunit is used to obtain the interaction event corresponding to the moving target speed, the rotating target angle and the moving target distance from the interactive event list as the target interaction event.

[0121] Furthermore, the three-dimensional coordinate calculation module 302 includes:

[0122] a target fingertip three-dimensional coordinate obtaining unit, used for obtaining the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip based on the fingertip three-dimensional coordinate set;

[0123] a fingertip distance calculation unit, configured to calculate the fingertip distance between the first target fingertip and the second target fingertip based on the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip;

[0124] The gesture action determination unit is used to determine that the gesture action is fingertip pinching when the fingertip distance is less than a second preset distance threshold, and obtain an interaction event corresponding to the fingertip pinching as the target interaction event.

[0125] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] The above-mentioned device can be implemented in the form of a computer program. Figure 4 Running on the headset shown.

[0127] See also Figure 4 , Figure 4 It is a schematic block diagram of the structure of a head-mounted device provided in an embodiment of the present application.

[0128] See also Figure 4 The head mounted device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0129] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any fingertip interaction method.

[0130] The processor is used to provide computing and control capabilities to support the operation of the entire head-mounted device.

[0131] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any fingertip interaction method.

[0132] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the head-mounted device to which the scheme of the present application is applied. The specific head-mounted device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0135] Based on a camera corresponding to the head mounted device, a hand image set of the user is collected, and each hand image in the hand image set is detected to obtain a fingertip two-dimensional coordinate set corresponding to the fingertip of the user;

[0136] Acquire camera parameters of the camera, and calculate a three-dimensional coordinate set of a fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip;

[0137] Based on the fingertip three-dimensional coordinate set, the user's gesture action and / or the movement trajectory of the user's fingertip in the spatial coordinate system are identified, and according to the gesture action and / or the movement trajectory, a target interaction event is determined to interact based on the interaction event.

[0138] In one embodiment, before the processor recognizes the user's gesture action and / or the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determines the target interaction event according to the gesture action and / or the motion trajectory, it is further used to implement:

[0139] Get the display content of the display interface;

[0140] Based on the displayed content, the interactive events corresponding to the displayed content and the fingertip motion trajectories corresponding to the interactive events are extracted from a preset interactive event database to generate an interactive event list for matching the target interactive events.

[0141] In one embodiment, the processor, when implementing the recognition of the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set and determining the target interaction event according to the motion trajectory, is used to implement:

[0142] Based on the fingertip three-dimensional coordinate set, acquiring change information of the fingertip three-dimensional coordinate, and based on the change information, identifying the movement trajectory of the user's fingertip;

[0143] An interaction event corresponding to the motion trajectory is obtained from the interactive event list as the target interaction event.

[0144] In one embodiment, the spatial coordinate system includes a depth coordinate axis, a horizontal coordinate axis, and a vertical coordinate axis. The processor recognizes the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determines the target interaction event according to the motion trajectory, further comprising:

[0145] Based on the fingertip three-dimensional coordinate set, identifying the change information of the user's fingertip in the spatial coordinate system, obtaining a first motion trajectory of the user's fingertip in the direction of the depth coordinate axis, a second motion trajectory of the user's fingertip in the longitudinal coordinate axis, and a third motion trajectory of the user's fingertip in the transverse coordinate axis;

[0146] An interaction event corresponding to at least one of the first motion trajectory, the second motion trajectory, and the third motion trajectory is obtained from the interactive event list as the target interactive event.

[0147] In one embodiment, the processor, when realizing identifying the user's gesture action and / or the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determining the target interaction event according to the gesture action and / or the motion trajectory, is further used to realize:

[0148] Based on at least one of the first motion trajectory, the second motion trajectory, and the third motion trajectory, identifying a movement parameter of the user's fingertip in the spatial coordinate system, the movement parameter including at least one of a movement speed, a rotation angle, and a movement distance;

[0149] The interactive event corresponding to the movement parameter is obtained from the interactive event list as the target interactive event.

[0150] In one embodiment, when the processor obtains the interactive event corresponding to the moving speed, the rotation angle, and the moving distance from the interactive event list as the target interactive event, it is configured to implement:

[0151] Compare a preset speed threshold, a preset angle threshold, and a first preset distance threshold with the moving speed, the rotation angle, and the moving distance, respectively, to obtain a comparison result;

[0152] According to the comparison result, the moving speed, rotation angle and moving distance that meet the preset exclusion condition are deleted to obtain the moving target speed, the rotation target angle and the moving target distance, wherein the preset exclusion condition includes that the moving speed is less than the speed threshold, the rotation angle is less than the preset angle threshold and the moving distance is less than the first preset distance threshold;

[0153] An interactive event corresponding to the moving target speed, the rotating target angle, and the moving target distance is obtained from the interactive event list as the target interactive event.

[0154] In one embodiment, the processor, when recognizing a user's gesture action based on the fingertip three-dimensional coordinate set and determining a target interaction event according to the gesture action, is used to implement:

[0155] Based on the fingertip three-dimensional coordinate set, acquiring the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip;

[0156] Calculating a fingertip distance between the first target fingertip and the second target fingertip based on the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip;

[0157] When the fingertip distance is less than a second preset distance threshold, the gesture action is determined to be fingertip pinching, and an interaction event corresponding to the fingertip pinching is acquired as the target interaction event.

[0158] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any fingertip interaction method provided in the embodiment of the present application.

[0159] The computer-readable storage medium may be an internal storage unit of the head-mounted device described in the foregoing embodiment, such as a hard disk or memory of the head-mounted device. The computer-readable storage medium may also be an external storage device of the head-mounted device, such as a plug-in hard disk, a smart memory card (Smart Medi a Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., provided on the head-mounted device.

[0160] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A fingertip interaction method, characterized in that: include: Based on a camera corresponding to the head mounted device, a hand image set of the user is collected, and each hand image in the hand image set is detected to obtain a fingertip two-dimensional coordinate set corresponding to the fingertip of the user; Acquire camera parameters of the camera, and calculate a three-dimensional coordinate set of a fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip; Based on the fingertip three-dimensional coordinate set, the user's gesture action and / or the movement trajectory of the user's fingertip in the spatial coordinate system are identified, and according to the gesture action and / or the movement trajectory, a target interaction event is determined to interact based on the interaction event.

2. The fingertip interaction method according to claim 1, characterized in that: Before the method of identifying the user's gesture action and / or the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set and determining the target interaction event according to the gesture action and / or the motion trajectory, the method further includes: Get the display content of the display interface; Based on the displayed content, the interactive events corresponding to the displayed content and the fingertip motion trajectories corresponding to the interactive events are extracted from a preset interactive event database to generate an interactive event list for matching the target interactive events.

3. The fingertip interaction method according to claim 2, characterized in that: The identifying the motion trajectory of the user's fingertip in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determining the target interaction event according to the motion trajectory, includes: Based on the fingertip three-dimensional coordinate set, acquiring change information of the fingertip three-dimensional coordinate, and based on the change information, identifying the movement trajectory of the user's fingertip; An interaction event corresponding to the motion trajectory is obtained from the interactive event list as the target interaction event.

4. The fingertip interaction method according to claim 2, characterized in that: The spatial coordinate system includes a depth coordinate axis, a horizontal coordinate axis, and a vertical coordinate axis. Based on the fingertip three-dimensional coordinate set, the motion trajectory of the user's fingertip in the spatial coordinate system is identified, and the target interaction event is determined according to the motion trajectory. The method also includes: Based on the fingertip three-dimensional coordinate set, identifying the change information of the user's fingertip in the spatial coordinate system, obtaining a first motion trajectory of the user's fingertip in the direction of the depth coordinate axis, a second motion trajectory of the user's fingertip in the longitudinal coordinate axis, and a third motion trajectory of the user's fingertip in the transverse coordinate axis; An interaction event corresponding to at least one of the first motion trajectory, the second motion trajectory, and the third motion trajectory is obtained from the interactive event list as the target interactive event.

5. The fingertip interaction method according to claim 4, characterized in that: The acquiring, from the list of interactive events, an interactive event corresponding to at least one of the first motion trajectory, the second motion trajectory, and the third motion trajectory as the target interactive event includes: Based on at least one of the first motion trajectory, the second motion trajectory, and the third motion trajectory, identifying a movement parameter of the user's fingertip in the spatial coordinate system, the movement parameter including at least one of a movement speed, a rotation angle, and a movement distance; The interactive event corresponding to the movement parameter is obtained from the interactive event list as the target interactive event.

6. The fingertip interaction method according to claim 5, characterized in that: Acquiring an interactive event corresponding to the moving speed, the rotation angle, and the moving distance from the interactive event list as the target interactive event includes: Compare a preset speed threshold, a preset angle threshold, and a first preset distance threshold with the moving speed, the rotation angle, and the moving distance, respectively, to obtain a comparison result; According to the comparison result, the moving speed, rotation angle and moving distance that meet the preset exclusion condition are deleted to obtain the moving target speed, the rotation target angle and the moving target distance, wherein the preset exclusion condition includes that the moving speed is less than the speed threshold, the rotation angle is less than the preset angle threshold and the moving distance is less than the first preset distance threshold; An interactive event corresponding to the moving target speed, the rotating target angle, and the moving target distance is obtained from the interactive event list as the target interactive event.

7. The fingertip interaction method according to any one of claims 1 to 6, characterized in that: The identifying the user's gesture based on the fingertip three-dimensional coordinate set, and determining the target interaction event according to the gesture, includes: Based on the fingertip three-dimensional coordinate set, acquiring the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip; Calculating a fingertip distance between the first target fingertip and the second target fingertip based on the three-dimensional coordinates of the first target fingertip and the three-dimensional coordinates of the second target fingertip; When the fingertip distance is less than a second preset distance threshold, the gesture action is determined to be fingertip pinching, and an interaction event corresponding to the fingertip pinching is acquired as the target interaction event.

8. A fingertip interaction device, characterized in that: include: A two-dimensional coordinate recognition module, used to collect a hand image set of the user based on a camera corresponding to the head mounted device, and detect each hand image in the hand image set to obtain a fingertip two-dimensional coordinate set corresponding to the user's fingertips; A three-dimensional coordinate calculation module, used for obtaining camera parameters of the camera, and calculating a three-dimensional coordinate set of the fingertip based on the camera parameters and the two-dimensional coordinate set of the fingertip; An interaction event determination module is used to identify the user's gesture actions and / or the movement trajectory of the user's fingertips in the spatial coordinate system based on the fingertip three-dimensional coordinate set, and determine the target interaction event according to the gesture actions and / or the movement trajectory, so as to interact based on the interaction event.

9. A head mounted device, characterized in that: The head mounted device comprises a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the fingertip interaction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the fingertip interaction method according to any one of claims 1 to 7.

Images