Pinch compensation for markers

Abstract

The present disclosure relates to pinch compensation for a marker. Generating a marker based on an input gesture includes obtaining hand tracking data of a hand based on one or more camera frames, and detecting a threshold change in a motion characteristic of the hand for a first frame that satisfies a threshold change. In response, determining whether a change in a pinch state is detected in a second frame within a threshold time of the first frame. In response to a change in the pinch state occurring within a threshold time period of the first frame, providing a hand position in the first frame for a user input action associated with the change in the pinch state from the second frame.

Description

Background Art

[0001] Some devices can generate and present extended reality (XR) environments. XR environments may include fully or partially simulated environments that people perceive and / or interact with via electronic systems. In XR, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with real-world properties. Some XR environments allow multiple users to interact with virtual objects or with each other in the XR environment. For example, a user can use hand gestures to perform a marking function. However, an improved technique is needed to manage the tracking of hands performing gestures to enhance marking. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1A to Figure 1B An example diagram illustrating the use of hand tracking to generate markers according to one or more embodiments is shown.

[0003] Figure 2 A flowchart of a technique for detecting input gestures directed to a marker according to some embodiments is shown.

[0004] Figure 3 A flow chart of a technique for initiating pinch compensation for a marker according to some embodiments is shown.

[0005] Figure 4 A diagram is shown for modifying a label using pinch compensation based on detected motion characteristics and changes in pinch state according to some embodiments.

[0006] Figure 5 A system diagram of an electronic device that can be used for gesture tagging according to one or more embodiments is shown.

[0007] Figure 6 Exemplary systems for various extended reality techniques are shown. DETAILED DESCRIPTION

[0008] The present disclosure relates to systems, methods, and computer-readable media for implementing gesture recognition and input. In some augmented reality scenarios, image data and / or other sensor data may be used to detect gestures by tracking hand data, which may then be converted into markers on a user interface (UI). The techniques described herein are intended to refine hand tracking data so that the resulting markers better capture user intent and adjust latency. Specifically, the techniques described herein relate to adjusting detected user input motions based on changes in pinch state, pinch progress, and / or motion characteristics to generate UI markers in order to account for missing data, latency, and other issues that may introduce inaccuracies into the resulting markers compared to the intended markers performed by the user.

[0009] When performing a markup gesture, the hand tracking feature may use sensor data capturing one or more of the user's hands to determine whether the hands are in a state where a markup is being generated. For example, pinching fingers together may indicate that a markup should be recorded, while the release of the pinch or "un-pinch" may indicate that the markup should be stopped. For example, problems may arise when a pinch or un-pinch event is detected too early or too late. This may result in artifacts being generated in the markup.

[0010] According to some embodiments, by considering not only the change of the pinch state (i.e., detecting a pinch event or an un-pinch event), but also the change of the motion characteristics of the hand, a mark can be generated more accurately. For example, hand tracking data can provide an estimate of the joints of the hand or hand (such as the pinch center of mass). According to some embodiments, the pinch center of mass of the hand can be determined to obtain user input, such as a mark. That is, the movement of the pinch center of mass can be converted into a mark on the UI. The motion characteristics of the pinch center of mass can be provided by hand tracking, or can be determined based on the position data of the pinch center of mass over a series of frames. In addition, in some embodiments, the motion characteristics of another joint in the hand, wrist, etc. can be used to determine the motion characteristics of the hand. If the change of the pinch state and the change of the motion characteristics (e.g., a sudden change in acceleration exceeds a predefined threshold and / or satisfies a threshold parameter, such as a change in direction, etc.) occur within a predefined time period of each other, the resulting mark can be modified. For example, if the change of the motion characteristics occurs before the pinch state changes from the un-pinch state to the pinch state, the mark corresponding to the frame between the change of the motion characteristics and the change of the pinch state can be removed. Similarly, if a change in motion characteristics occurs before the pinch state changes from a pinched state to an unpinched state, the location of the pinch centroid in the frames between the change in motion characteristics and the change in pinch state can be used to add a marker. In other words, if the change in pinch state occurs near a substantial change in motion characteristics, the point at which the marker begins or ends is modified to coincide with the point at which the change in motion characteristics was detected.

[0011] In the following disclosure, a physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset or representation of a person's physical motion is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphics content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment and adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0012] There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0013] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this description, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed concept. For the sake of clarity, all features of the actual specific implementation may not be described. In addition, as part of this specification, some drawings of the present disclosure may be provided in the form of flow charts. The blocks in any particular flow chart may be presented in a specific order. However, it should be understood that the specific order of any given flow chart is only used to illustrate an embodiment. In other embodiments, any element in the various elements depicted in the flow chart may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. In addition, the language used in the present disclosure has been selected primarily for readability and guidance purposes, and may not be selected to delimit or limit the subject matter of the present invention, or to resort to the necessary claims to determine such subject matter of the invention. Reference in this disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.

[0014] It should be understood that in the development of any actual implementation (as in any software and / or hardware development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary from one implementation to another. It should also be understood that such development work may be complex and time-consuming, but is nonetheless a routine task for those of ordinary skill in the design and implementation of a graphical modeling system having the benefit of this disclosure.

[0015] For purposes of this application, the term "markup" refers to a drawing or other enhancement to a user interface that corresponds to a stroke performed by a user in the form of a persistent gesture.

[0016] Figure 1AA diagram of a technique for generating a marker using a pinch gesture according to one or more embodiments is shown. Specifically, FIG. 1 depicts an example of a technique for converting a gesture into a UI marker based on a detected pinch event but without regard to motion characteristics. Gesture 100 includes a series of hand postures that constitute a pinch marker input. The selection of postures shown may represent hand postures detected at corresponding times, such as postures captured in a camera frame at corresponding times. Initially, at T1 102A, the hand is not pinched. Similarly, at T2 102B, the hand remains unpinched, but changes direction. At T3 102C, the hand posture is shown as being in a pinched position. Therefore, the raw hand tracking data 110 includes a first segment 112, in which the hand is determined to be in an unpinched state, and therefore no marker is generated.

[0017] Between T3 102C and T4 102D, the gesture 100 includes an arcuate motion during which the hand is detected to be in a pinch state. At T4 102D, the hand changes direction again from an upward motion to a downward motion. T5 102E is the last frame in which the hand is identified as being in a pinch state. Therefore, the raw hand tracking data 110 includes a second segment 114 between T3 102C and T5 102E in which the hand is determined to be in a pinch state. Because the pinch is determined to be in a pinch state, a mark can be generated. In some embodiments, the result mark reflects the movement of the hand or a portion of the hand when it is in the pinch state. For example, the pinch center of mass can be determined based on a specific positional relationship with the hand. For example, the pinch center of mass can be at a location where two fingers (such as an index finger and a thumb) are in contact, or it can be determined in another way-such as an offset from the hand origin or other location. In some embodiments, a mark can be generated to reflect the movement of the pinch center of mass.

[0018] At T6 102F, the hand is determined to be in an unpinched state. The raw hand tracking data 110 additionally includes a third segment 116 in which the hand is determined to be in an unpinched state and therefore the stop is marked.

[0019] Display 120 depicts an example display device of an electronic device in which a mark generated by gesture 100 is presented. Display 120 can be a part of a multifunctional electronic device such as a mobile device or wearable device or other electronic device including a display capable of presenting UI marks. Raw data UI marks 122 depict marks on the UI derived from raw hand tracking data 110, taking into account the pinch state but not the motion characteristics. As shown, the shape of the mark is aligned with the motion of gesture 100 during the second segment 114, and includes an arc captured between T3 102C and T4 102D and a change in direction between T4 102D and T5 102E.

[0020] However, because the threshold change in the motion characteristics of the hand at T2 (here, a change in speed) was captured just before the pinch press at T3 102C, it is possible that the pinch press at T3 102C was a late detected pinch, and the user intended to start marking at T2 102B, where the threshold change in motion characteristics occurred. Similarly, the motion between T4 102D and T5 102E is captured as a mark, but the change in motion characteristics at T4 102D occurs just before the change in the pinch state at T5 102E. As a result, the resulting raw data UI markup 122 includes a hook element at the end of the markup, which may be unintentional and due to wait time, user error, etc.

[0021] Figure 1A The bar graph 140 shows an example determined velocity change at each time identified in the motion of the hand gesture 100. In particular, the bar graph 140 shows the change in velocity of the hand in the form of acceleration measurements at each identified time compared to a threshold motion characteristic in the form of a threshold acceleration 142. Thus, a threshold change in motion characteristic is detected at T2 102B and T4 102D.

[0022] Steering Figure 1B , presents an example diagram in which a marker is modified based on adjusted hand tracking data according to one or more embodiments. Specifically, Figure 1B Described is an example of a technique for converting a gesture into a UI marker based on a detected pinch event and taking into account motion characteristics. Figure 1A As described, the raw hand tracking data 110 includes a first segment 112, in which the hand is determined to be in an unpinched state, and therefore no marking is generated. In addition, the raw hand tracking data 110 includes a second segment 114, in which the hand is determined to be in a pinched state, and markings corresponding to hand movements can be generated. In addition, the raw hand tracking data 110 includes a third segment 116, in which the hand is determined to be in an unpinched state, and therefore marking stops.

[0023] According to one or more embodiments, the marker may be adjusted based on a determination that a change in the pinch state occurs within a predetermined time and a change in the motion characteristic satisfies a threshold. Figure 1A, because the motion characteristic at T2 102B was captured just before the pinch press at T3 102C, it is possible that the pinch press at T3 102C was a late detected pinch, and the user intended to start marking at T2 102B, at which the change in motion characteristic occurred. Similarly, the motion between T4 102D and T5 102E is captured as marking, as shown with a downward hook at the end of the raw data UI marker 122, but the change in motion characteristic at T4 102D occurs just before the change in the pinch state at T5 102E, indicating that the intended marking action starts at T4 102D.

[0024] Out of Figures 1A to 1B For purposes of the example, it may be determined that a change in the motion characteristic at T2 102B and a change in the motion characteristic at T4 102D each satisfy a threshold change in the motion characteristic that is monitored to determine whether to adjust the marker. Additionally, it may be determined that the change in the motion characteristic at T2 102B and the detected pinch press at T3 102C occur within a predefined time period. According to one or more embodiments, the determination that the change in the motion characteristic satisfies the threshold motion characteristic and that a change in the pinch state occurs within a predefined time period indicates that the resulting marker should be adjusted. If a change in the pinch state is detected after a change in the motion characteristic, it may be determined that the pinch was detected later, or that the user inadvertently pinched later. Thus, the system may determine that the resulting marker should be adjusted to capture missed hand movements. As shown herein, the motion from the hand tracking data is adjusted at the first modification segment 152 to indicate a touch. Although in Figure 1B , the adjustments are depicted as being applied to the hand tracking data (e.g., the original hand tracking data 110 compared to the adjusted hand tracking data 150), but in some embodiments, adjustments may be made to the marker itself (e.g., the original data UI marker 122), such as the adjusted UI marker 160 shown. As a result, the resulting adjusted UI marker 160 now starts earlier than the original data UI marker 122. In some embodiments, the adjusted UI marker 160 is adjusted by obtaining position information of the hand, such as position information of the pinch centroid for frames between a threshold change in velocity and a change in the pinch state. The position information is used to generate portions of the marker that may not have been initially captured based on the original hand tracking data. For example, the position of the hand can be converted into a marker for presentation on the display 120. Thus, in some embodiments, initially, the portion of the marker corresponding to the second segment 114 can be displayed before the portion of the marker associated with the first modified segment 152.

[0025] Additionally, it may be determined that the change in motion characteristics at T4 102D occurs within a predefined time period of the change in pinch state at T5 102E. According to one or more embodiments, because the change in pinch state is detected after the change in motion characteristics, it may be determined that an un-pinch event was detected later, or that the user un-pinched later without intending to. For example, the user may have intended to end the marking and abruptly drop their hand, but the un-pinch occurred during the drop rather than at the start of the drop. Thus, the system may determine that the resulting marking should be adjusted to remove the additional captured movement of the hand. As shown herein, the motion from the hand tracking data is adjusted at the second modification segment 154 to indicate that a pinch did not occur. Although in Figure 1B , the adjustments are depicted as being applied to the hand tracking data (e.g., the original hand tracking data 110), but in some embodiments, adjustments may be made to the marker itself (e.g., the original data UI marker 122), such as the adjusted UI marker 160 shown. As a result, the resulting adjusted UI marker 160 now ends earlier than the initial original data UI marker 122, as is apparent from the lost hooking feature at the end of the adjusted UI marker 160. In some embodiments, the adjusted UI marker 160 is adjusted by obtaining position information of the hand, such as position information of the pinch centroid for frames between a threshold change in speed and a change in the pinch state. This position information is used to remove the portion of the marker that was initially determined to be used for marking based on the pinch state without regard to changes in motion characteristics. Thus, in some embodiments, initially, the portion of the marker corresponding to the second modification segment 154 may be displayed before the associated marker is removed. In some embodiments, changes may be used to adjust the marker. For example, if the user generally moves slowly, the threshold change in motion characteristics may be different than if the user generally moves faster. As another example, if the user's motion is generally below a threshold speed, pinching and un-pinching may be considered more precise and the markers may not be adjusted.

[0026] Figure 2 A flow chart of a technique for generating a mark using an input gesture such as a pinch according to some embodiments is shown. In particular, Figure 2 A flow chart of a method for generating a mark based on hand movement and detected pinching without considering threshold changes in motion characteristics is shown. For the purpose of explanation, the following steps will be described as being performed by specific components. However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions can be added.

[0027] Flowchart 200 begins at box 205, where hand tracking data is obtained from a camera frame or other sensor data frame. The hand tracking data may include, for example, image data, depth data, etc., from which hand posture, position, and / or motion can be estimated. In some embodiments, the hand tracking data may include or be based on additional sensor data, such as image data and / or depth data capturing one or more hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. In some embodiments, the sensor data may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data of the electronic device, such as the position and orientation of the device.

[0028] Flowchart 200 continues at box 210, where a determination is made as to whether a pinch is detected. Pinch detection may rely on a hand pose determination based on hand tracking data. In some embodiments, the hand pose may be estimated based on a predicted position of one or more joints of the hand. For example, a hand tracking network may provide an estimated position of each joint in a set of joints of the hand. In some embodiments, the hand pose may be determined based on a relative position of a set of these joints or a subset of these joints. Additionally or alternatively, the gap distance between two fingers (such as an index finger and a thumb) may be used to determine whether a pinch occurs. The gap distance may be measured in the sensor data or predicted based on the sensor data. When it is determined that the gap distance is determined to be within a predetermined threshold, and / or if the hand pose satisfies the pinch parameter, a pinch press event is determined to have occurred, such as "pinch press". If a pinch event is not detected at box 210, the flowchart proceeds to box 215, and the system continues to receive hand tracking data without generating a marker.

[0029] Returning to box 210, if it is determined that a pinch event is detected, flowchart 200 proceeds to box 220. At box 220, a mark is generated on the UI based on the hand movement. According to one or more embodiments, the mark is generated by applying digital or virtual ink in a manner consistent with the travel of the hand when the marking gesture is active. For example, when a pinch is detected, the travel or movement of the hand or a part of the hand will be tracked and converted into a mark on the user interface. According to one or more embodiments, the mark is generated based on the position of a part of the hand during the input gesture. According to one or more embodiments, the pinch centroid is a point in space that is tracked to determine user input using a pinch. The pinch centroid can be determined based on characteristics of the position and orientation of the hand. For example, based on the posture of the hand, the pinch centroid can be determined at a specific offset including a distance and angle from a specific position on the hand.

[0030] As additional frames of sensor data are received, markers are generated based on the corresponding hand positions. From this, flowchart 200 proceeds to block 225, where a determination is made as to whether a pinch release is detected (e.g., a "pinch up" event). According to one or more embodiments, the determination as to whether a pinch release is detected can be based on the predicted gap distance exceeding a predefined threshold, and / or a touch between two fingers (such as an index finger and a thumb) is no longer predicted. When a pinch release is not detected, flowchart 200 returns to block 220, and the system continues to generate markers corresponding to the user's hand positions.

[0031] Returning to block 225, if a determination is made that a pinch release is detected, the flowchart proceeds to block 230 and the system stops marking on the UI based on the hand movement. That is, the hand movement is no longer used to generate markings on the UI. The flowchart 200 then returns to block 215 and the system continues to receive hand tracking data until the next pinch event is detected.

[0032] Steering Figure 3 , shows a flow chart of a technique for generating a markup using an input gesture according to some embodiments. In particular, Figure 3 A flow chart of a method for generating a marker based on hand movement and detected pinching while taking into account threshold changes in motion characteristics is shown. Figure 2 The flowchart described here is used to perform Figure 3 Flowchart of . For the purpose of explanation, the following steps will be described as being performed by specific components. However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions can be added.

[0033] Flowchart 300 begins at box 305, where hand tracking data is obtained from a camera frame or other sensor data frame. The hand tracking data may include, for example, image data, depth data, etc., from which hand posture, position, and / or motion can be estimated. In some embodiments, the hand tracking data may include or be based on additional sensor data, such as image data and / or depth data capturing one or more hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. In some embodiments, the sensor data may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data of the electronic device, such as the position and orientation of the device.

[0034] Flowchart 300 continues at block 310 where a determination is made as to whether the motion characteristic of the hand satisfies a threshold change in the motion characteristic. As described above, a change in the motion characteristic that occurs in the vicinity of a change in the pinch state may be more indicative of an intent to start or end a tag than a change in the pinch state alone. The motion characteristic may indicate one or more differences in the motion characteristic, such as a sudden acceleration, deceleration, change in direction, etc., that satisfy a threshold change in the motion characteristic that indicates the user's intent to start or end a tag. Examples of changes in the motion characteristic may include, for example, a sudden lowering of the hand, a "hook-back" motion where the hand changes direction quickly, etc. Other examples include a sudden acceleration of the hand exceeding a predefined threshold acceleration, a deceleration below a predefined threshold deceleration, etc.

[0035] According to one or more embodiments, the determination of the change in the motion characteristic at block 310 may be similar to the determination of the change in the motion characteristic at block 310 as described above with respect to Figure 2 The determination that a pinch event is detected may occur synchronously or asynchronously as described in block 210 of FIG. That is, a particular frame may be analyzed to determine whether a pinch event is detected based on the current frame and / or one or more additional frames. In some embodiments, the same frame is then analyzed to determine a change in the motion characteristic, for example using the same set of frames used to detect the pinch event or an alternate set of frames. Alternatively, the determination that the change in the motion characteristic satisfies the speed parameter may occur later in the pipeline such that Figure 3 The method depicted in occurs in response to detecting a change in the pinch state. If no velocity change is detected at block 310 that satisfies the threshold change, the flowchart proceeds to block 315 and the system continues to receive hand tracking data.

[0036] Returning to box 310, if the change in motion characteristics does not meet the threshold change, the flowchart 300 continues to box 320, where a determination is made as to whether a change in the pinch state is detected within a threshold time period of the change in motion characteristics from box 310. As described above, according to one or more embodiments, a change in the pinch state that is in close proximity to the detected change in motion characteristics but not occurring simultaneously may indicate that a pinch or no pinch is detected early or late. Therefore, the threshold time period may include a time period before and / or after the detected change in the pinch state. In some embodiments, the threshold time period may be constant, or may be dynamic based on user preferences, contextual factors, etc. For example, in some embodiments, the threshold time period may be application specific. In addition, the threshold time period may be dynamically modified based on other factors (such as hand speed, other components on the user interface, etc.). If it is determined at box 320 that a change in the pinch state is not detected within the threshold time period, the flowchart proceeds to box 315, and the system continues to receive hand tracking data.

[0037] If it is determined at box 320 that a change in the pinch state is detected within a threshold time period, flowchart 300 ends at box 325. At box 325, the mark is modified based on the hand tracking data from the point where the threshold change in the motion characteristic is detected. For example, if a pinch press is detected just before the threshold change in the motion characteristic and within the threshold time period, the corresponding mark can be modified to remove the mark generated in association with the frame of the hand tracking data between the detected pinch press and the detected threshold change motion characteristic. That is, the pinch can be determined to occur earlier than the user intended, resulting in excessive marking. As another example, if a pinch press is detected within a predefined time period of the threshold change in the motion characteristic, the corresponding mark can be modified based on the hand tracking data captured in the frame between the detected threshold change in the motion characteristic and the change in the pinch state to add the mark to the UI. That is, it can be determined that the pinch press has been detected later, resulting in the loss of the mark.

[0038] Similarly, if a pinch-up is detected just prior to a change in motion characteristics and within a threshold time period, the corresponding mark can be modified to add a mark generated in association with the frame of hand tracking data between the detected pinch-up and the detected change in motion characteristics. That is, the pinch-up can be determined to have occurred earlier than the user intended, resulting in lost marking. As another example, if a pinch-up is detected just after a change in motion characteristics and within a predefined time period, the corresponding mark can be modified to remove the mark in the UI based on the hand tracking data captured in the frame between the detected change in motion characteristics and the change in pinch state. That is, it can be determined that the pinch-up has been detected late, resulting in excessive marking.

[0039] Figure 4 A diagram is shown for modifying a marker based on motion characteristics and pinch state according to some embodiments. Specifically, Figure 4 More details are given on how to adjust the marker based on the change of pinch state and the threshold change of motion characteristics, for example as described above in Figure 3 325 of . For the purpose of explanation, the following steps will be described as being performed by specific components. However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0040] Flowchart 400 begins at box 405, where the hand position is determined for the first frame in which a change in a motion characteristic that satisfies a threshold change in the motion characteristic is detected. According to one or more embodiments, the hand position may be determined in a 2D or 3D coordinate system by an electronic device using tracking technology. In some embodiments, hand tracking data captured by a user-wearable device such as a head-mounted device may be used to determine the hand position. In some embodiments, sensor data is captured corresponding to the movement and characteristics of the hand and fed into a hand tracking pipeline that provides data related to the position of the hand, the posture of the hand, and the like. The position of the hand may be provided, for example, in a user-centric coordinate system, and the movement through the coordinate system may be converted to generate UI markers.

[0041] The flowchart continues to box 410, where the hand position is determined for the second frame in which the change in the pinch state is detected. According to one or more embodiments, the change in the pinch state may be determined by the hand tracking pipeline using the same sensor data or different sensor data as used to determine the hand position. Although boxes 405 and 410 are depicted in a specific order, it should be understood that in some embodiments, the frame corresponding to the change in the pinch state may be temporarily captured before the frame corresponding to the threshold change in the motion characteristic. In addition, although the flowchart describes the frames as "first" and "second" frames, it should be understood that this designation only indicates that the frames are two different frames. In addition, the first frame and the second frame may or may not be consecutive frames. That is, the first frame may be captured at a time before the second frame, or the second frame may be captured at a time before the first frame.

[0042] At box 415, determine whether a change in the pinch state is detected after a threshold change in the motion characteristic. That is, determine whether the second frame from box 410 is temporarily captured after the first frame from box 405. If it is determined at box 415 that a change in the pinch state is detected after a threshold change in the motion characteristic, the flowchart continues to box 420, where the frame between the first frame at box 405 and the second frame at box 410 identifies the historical hand position. According to one or more embodiments, a record of the hand position of each frame is stored. In other embodiments, the historical hand position can be determined for a previous frame based on historical hand tracking data. Thus, the system can refer to the historical hand position of each frame between the first frame and the second frame. In some embodiments, the hand position can be determined in a coordinate system associated with a user, a device, etc. The flowchart then ends at box 425, where a mark is adjusted on the UI based on the historical hand position. That is, if the change in pinch state involves a change from pinched to unpinched, and the change in pinch state is detected after a threshold change in the motion characteristic, the historical hand positions identified at block 420 may be referenced against the resulting marks to remove marks corresponding to those hand positions. Similarly, if the change in pinch state involves a change from unpinched to pinched, and the change in pinch state is detected after a threshold change in the motion characteristic, the historical hand positions identified at block 420 may be referenced against the resulting marks to add marks corresponding to those hand positions.

[0043] Returning to block 415, if a determination is made that a change in the pinch state is not detected after the threshold change in the motion characteristic, the flowchart continues to block 430, where a determination is made as to whether a change in the pinch state is detected before the threshold change in the motion characteristic. Although decision blocks 415 and 430 are shown in a particular order, it should be understood that the decisions may be made in a different order or simultaneously.

[0044] If it is determined at box 430 that a change in the detected pinch state occurs before a threshold change in the motion characteristic, the flowchart proceeds to box 435, where historical hand positions are identified from frames between the first frame at box 405 and the second frame at box 410. According to one or more embodiments, a record of the hand position of each frame is stored. In other embodiments, the historical hand position may be determined for a previous frame based on historical hand tracking data. Thus, the system may refer to the historical pinch position of each frame between the first frame and the second frame. In some embodiments, the hand position may be determined in a coordinate system associated with a user, a device, etc. The flowchart then ends at box 440, where a mark is adjusted on the UI based on the historical hand position. That is, if the change in the pinch state involves a change from pinched to un-pinched, and the change in the pinch state is detected before a threshold change in the motion characteristic, the historical hand position identified at box 420 may be used to retroactively add additional marks to the marks that have been generated or determined to be generated on the UI corresponding to those hand positions. Similarly, if the change in pinch state involves a change from pinched to unpinched, and the change in pinch state is detected after a threshold change in motion characteristics, the historical hand positions identified at box 420 can be referenced against the resulting markers to remove markers corresponding to those hand positions.

[0045] Returning to box 430, if it is determined that a change in the pinch state was not detected before the threshold change in the motion characteristic, the flowchart ends at box 445 and no changes are made to the tag. That is, when the pinch state change and the threshold change in the motion characteristic occur shortly before or after each other, the tag is modified. If a determination is made at box 430 that a change in the pinch was not detected before the threshold change in the motion characteristic, and a determination is made at box 415 that a change in the pinch was not detected after the threshold change in the motion characteristic, the two events may occur at the same or similar time. When the change in the pinch state and the threshold change in the motion characteristic are determined to occur at the same or similar time, the flowchart ends at box 445 and no changes are made to the tag. That is, when the change in the pinch state and the threshold change in the motion characteristic are determined to occur at the same or similar time, it can be determined that the user's intent matches the tag.

[0046] refer to Figure 5, depicting a simplified block diagram of an electronic device 500. The electronic device 500 may be part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted system, a projection-based system, a base station, a laptop computer, a desktop computer, a network device, or any other electronic system such as those described herein. The electronic device 500 may include one or more additional devices, such as a server device, a base station, an auxiliary device, etc., and various functions may be included in the one or more additional devices or various functions may be distributed on the one or more additional devices. Illustrative networks include, but are not limited to, local networks (such as universal serial bus (USB) networks), local area networks of organizations, and wide area networks (such as the Internet). According to one or more embodiments, the electronic device 500 is used to interact with the user interface of the application 535. According to one or more embodiments, the application 535 may include one or more editing applications, or otherwise provide editing functions such as marked applications. It should be understood that the various components and functions within the electronic device 500 may be distributed differently on modules or components or even additional devices.

[0047] The electronic device 500 may include one or more processors 520, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 500 may also include a memory 530. The memory 530 may include one or more different types of memories, which may be used to perform device functions in conjunction with the processor 520. For example, the memory 530 may include a cache, a ROM, a RAM, or any type of transient or non-transient computer-readable storage medium capable of storing computer-readable code. The memory 530 may store various programming modules executed by the processor 520, including a tracking module 545 and various other applications 535. The electronic device 500 may also include a storage device 540. The storage device 540 may include one or more non-transient computer-readable media, including, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). The storage device 540 may be used to store various data and structures, which may be used to store data related to hand tracking and UI preferences. According to one or more embodiments, storage device 540 may be configured to store hand tracking network 555 and other data used to determine hand motion, such as registration data 550. The electronic device may additionally include a network interface from which electronic device 500 can communicate over a network.

[0048] The electronic device 500 may also include one or more cameras 505 or other sensors 510, such as a depth sensor from which the depth of the scene can be determined. In one or more embodiments, each of the one or more cameras 505 can be a traditional RGB camera or a depth camera. In addition, the camera 505 can include a stereo camera or other multi-camera system. In addition, the electronic device 500 may include other sensors that can collect sensor data for tracking user movement, such as a depth camera, an infrared sensor, or an orientation sensor, such as one or more gyroscopes, accelerometers, etc.

[0049] According to one or more embodiments, the memory 530 may include one or more modules including computer readable code that can be executed by the processor 520 to perform functions. The memory 530 may include, for example, a tracking module 545 and one or more applications 535. The tracking module 545 can be used to track the position of the hand and other user movements in the physical environment. The tracking module 545 can use sensor data, such as data from the camera 505 and / or the sensor 510. In some embodiments, the tracking module 545 can track user movement to determine whether to trigger user input according to the input gesture detected, such as a marking function. The electronic device 500 may also include a display 525 that can present a UI (such as presenting a mark generated by a tracking gesture) for user interaction. For example, the UI may be associated with one or more applications in the application 535. The display 525 may be an opaque display or may be translucent or transparent. The display 525 may be incorporated with LEDs, OLEDs, digital light projectors, silicon-based liquid crystals, etc.

[0050] Although electronic device 500 is depicted as including the above-described multiple components, in one or more embodiments, various components may be distributed across multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the differently distributed functionality. In addition, additional components may be used, and certain combinations of the functionality of any component may also be combined.

[0051] Reference now Figure 6, a simplified functional block diagram of an exemplary multifunctional electronic device 600 according to one embodiment is shown. Each of the electronic devices may be a multifunctional electronic device, or may have some or all of the components described in the multifunctional electronic device described herein. The multifunctional electronic device 600 may include a processor 605, a display 610, a user interface 615, graphics hardware 620, a device sensor 625 (e.g., a proximity sensor / ambient light sensor, an accelerometer and / or a gyroscope), a microphone 630, an audio codec 635, a speaker 640, a communication circuit 645, a digital image capture circuit 650 (e.g., including a camera system), a video codec 655 (e.g., supporting a digital image capture unit), a memory 660, a storage device 665, and a communication bus 670. The multifunctional electronic device 600 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0052] The processor 605 can execute instructions necessary for implementing or controlling the operation of many functions performed by the device 600 (such as, the generation and / or processing of images as disclosed herein). The processor 605 can, for example, drive the display 610 and can receive user input from the user interface 615. The user interface 615 can allow the user to interact with the device 600. For example, the user interface 615 can take a variety of forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, touch screens, sight lines and / or gestures. For example, the processor 605 can also be a system on a chip, such as those found in mobile devices, and include a dedicated GPU. The processor 605 can be based on a reduced instruction set computer (RISC) or a complex instruction set computer (CISC) architecture or any other suitable architecture, and can include one or more processing cores. The graphics hardware 620 can be a dedicated computing hardware for processing graphics and / or assisting the processor 605 to process graphics information. In one embodiment, the graphics hardware 620 may include a programmable GPU.

[0053] Image capture circuit 650 may include two (or more) lens assemblies 680A and 680B, each of which may have a separate focal length. For example, lens assembly 680A may have a shorter focal length relative to the focal length of lens assembly 680B. Each lens assembly may have a separate associated sensor element 690A and 690B. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuit 650 may capture static images and / or video images. The output from image capture circuit 650 may be processed by video codec 655 and / or processor 605 and / or graphics hardware 620 and / or a dedicated image processing unit or pipeline incorporated into circuit 650. Such captured images may be stored in memory 660 and / or storage device 665.

[0054] The sensor and camera circuit 650 can capture still images and video images that can be processed at least in part by the following devices according to the present disclosure: video codec 655 and / or processor 605 and / or graphics hardware 620 and / or a dedicated image processing unit incorporated into the circuit 650. The captured images can be stored in memory 660 and / or storage device 665. Memory 660 may include one or more different types of media used by processor 605 and graphics hardware 620 to perform device functions. For example, memory 660 may include memory cache, read-only memory (ROM) and / or random access memory (RAM). Storage device 665 can store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage device 665 may include one or more non-transitory computer-readable storage media, including, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and DVDs), and semiconductor memory devices (such as EPROMs and EEPROMs). The memory 660 and storage device 665 can be used to tangibly hold computer program instructions or codes organized into one or more modules and written in any desired computer programming language. For example, when executed by the processor 605, such computer program code can implement one or more of the methods described herein.

[0055] The various processes defined herein contemplate the option of obtaining and utilizing a user's identifying information. For example, such personal information may be utilized in order to track a user's movements. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent, and the user should be aware of and control the use of their personal information.

[0056] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive and meet or exceed government / industry standards. In addition, parties may not distribute, sell or otherwise share such information except for any reasonable and lawful purpose.

[0057] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and data deletion. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth), controlling the amount or characteristics of stored data (e.g., collecting location data at the city level rather than the address level), controlling how the data is stored (e.g., aggregating data across users), and / or other methods.

[0058] It should be understood that the above description is intended to be illustrative and not limiting. The material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims, and the material is provided in the context of specific embodiments, variations of which will be apparent to those skilled in the art (for example, some of the disclosed embodiments may be used in conjunction with each other). Therefore, Figures 2 to 4 The specific arrangement of steps or actions shown in or in FIG. Figures 5 and 6 The arrangement of elements shown in the accompanying drawings should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein".

Claims

1. A method, comprising: obtaining hand tracking data for the hand based on the one or more camera frames; detecting at a first frame that a change in a motion characteristic of the hand satisfies a threshold change in the motion characteristic; Detecting a change in the gesture state at a second frame; determining that the second frame is within a threshold time period of the first frame; as well as In response to the determination, a hand position for a user input action associated with the change in gesture state is adjusted according to the hand position at the first frame. 2 . The method according to claim 1 , wherein the user input action is a marking action, and the marking action includes adding or removing a mark on a user interface according to the change of the motion characteristic.

3. The method of claim 2, wherein the first frame is captured before the second frame, and wherein the change in gesture state comprises a transition from pinched to unpinched, the method further comprising, in response to the determining: A marker on the user interface is removed from a location on the user interface associated with the hand position in the first frame to a location on the user interface associated with the hand position in the second frame.

4. The method of claim 2, wherein the first frame is captured after the second frame, and wherein the change in gesture state comprises a transition from non-pinch to pinch, the method further comprising, in response to the determining: A marker on the user interface is removed from a location on the user interface associated with the hand position in the first frame to a location on the user interface associated with the hand position in the second frame.

5. The method according to any one of claims 3 to 4, wherein removing the mark comprises: obtaining historical hand positions of frames captured between the first frame and the second frame; as well as A marker is removed on the user interface based on the historical hand position.

6. The method of claim 2, wherein the first frame is captured after the second frame, and wherein the change in gesture state comprises a transition from pinched to unpinched, the method further comprising, in response to the determining: A marker is added on the user interface between a location on the user interface associated with the hand position in the second frame and a location on the user interface associated with the hand position in the first frame.

7. The method of claim 2, wherein the first frame is captured before the second frame, and wherein the change in gesture state comprises a transition from non-pinch to pinch, the method further comprising, in response to the determining: A marker is added on the user interface between a location on the user interface associated with the hand position in the second frame and a location on the user interface associated with the hand position in the first frame.

8. The method according to any one of claims 6 to 7, wherein adding a mark comprises: obtaining historical hand positions of frames captured between the first frame and the second frame; as well as The mark is added on the user interface according to the historical hand position.

9. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to: obtaining hand tracking data for the hand based on the one or more camera frames; detecting at a first frame that a change in a motion characteristic of the hand satisfies a threshold change in the motion characteristic; Detecting a change in the gesture state at a second frame; determining that the second frame is within a threshold time period of the first frame; as well as In response to the determination, a hand position for a user input action associated with the change in gesture state is adjusted according to the hand position at the first frame. 10 . The non-transitory computer-readable medium of claim 9 , wherein the user input action is a marking action, the marking action comprising adding or removing a mark on a user interface according to the change of the motion characteristic.

11. The non-transitory computer readable medium of claim 10, wherein the first frame is captured after the second frame, and wherein the change in gesture state comprises a transition from non-pinch to pinch, and the non-transitory computer readable medium further comprises computer readable code for, in response to the determining: A marker on the user interface is removed from a location on the user interface associated with the hand position in the first frame to a location on the user interface associated with the hand position in the second frame.

12. The non-transitory computer readable medium of claim 10, wherein the first frame is captured after the second frame, and wherein the change in gesture state comprises a transition from pinched to un-pinched, and the non-transitory computer readable medium further comprises computer readable code to, in response to the determining: A marker is added on the user interface between a location on the user interface associated with the hand position in the second frame and a location on the user interface associated with the hand position in the first frame.

13. The non-transitory computer readable medium of claim 10, wherein the first frame is captured before the second frame, and wherein the change in gesture state comprises a transition from non-pinch to pinch, and the non-transitory computer readable medium further comprises computer readable code for, in response to the determining: A marker is added on the user interface between a location on the user interface associated with the hand position in the second frame and a location on the user interface associated with the hand position in the first frame.

14. The non-transitory computer readable medium of any one of claims 12 to 13, wherein the computer readable code for adding a tag comprises computer readable code for: obtaining historical hand positions for frames captured between the first frame and the second frame; and The mark is added on the user interface according to the historical hand position.

15. A system, comprising: one or more processors; and One or more computer readable media comprising computer readable code executable by the one or more processors to: obtaining hand tracking data for the hand based on the one or more camera frames; detecting at a first frame that a change in a motion characteristic of the hand satisfies a threshold change in the motion characteristic; Detecting a change in the gesture state at a second frame; determining that the second frame is within a threshold time period of the first frame; as well as In response to the determination, a hand position for a user input action associated with the change in gesture state is adjusted according to the hand position at the first frame. 16 . The system of claim 15 , wherein the user input action is a marking action, the marking action comprising adding or removing a mark on a user interface according to the change of the motion characteristic.

17. The system of claim 16, wherein the first frame is captured before the second frame, and wherein the change in gesture state comprises a transition from pinched to un-pinched, and the system further comprises computer readable code to, in response to the determining: A marker on the user interface is removed from a location on the user interface associated with the hand position in the first frame to a location on the user interface associated with the hand position in the second frame.

18. The system of any one of claims 16 to 17, wherein the computer readable code for removing the marking comprises computer readable code for: obtaining historical hand positions for frames captured between the first frame and the second frame; and A marker is removed on the user interface based on the historical hand position.

19. The system of claim 16, wherein the first frame is captured after the second frame, and wherein the change in gesture state comprises a transition from pinched to unpinched, and the system further comprises computer readable code to, in response to the determining: A marker is added on the user interface between a location on the user interface associated with the hand position in the second frame and a location on the user interface associated with the hand position in the first frame.

20. The system of claim 16, wherein the first frame is captured before the second frame, and wherein the change in gesture state comprises a transition from non-pinch to pinch, and the system further comprises computer readable code to: in response to the determining: A marker is added on the user interface between a location on the user interface associated with the hand position in the second frame and a location on the user interface associated with the hand position in the first frame.

Images