Easily handled body-based AR system input

By using computer vision-based hand tracking technology in head-mounted augmented reality devices to identify and analyze user's hand postures, the problem of limited user input modes in the prior art is solved, and more natural and intuitive user interaction is achieved, and operation efficiency and experience quality are improved.

CN120035804APending Publication Date: 2025-05-23SNAP INC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202380072554.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-10-12
Filing Date
2023-10-05
Publication Date
2025-05-23

AI Technical Summary

Technical Problem

The user input modes of existing head-mounted augmented reality devices are limited, making it difficult to achieve complex user interactions, especially in the absence of physical input devices for touch screens or keyboards.

Method used

Using computer vision-based hand tracking technology, it provides a richer user input mode by detecting and tracking the user's hand posture and movement. Specific implementations include using a hand tracking platform and posture component framework to identify and parse the user's hand posture in order to convert it into input commands for the user interface.

Benefits of technology

It realizes more natural and intuitive user interaction in an augmented reality environment, improving the user's operational efficiency and experience quality in the AR system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120035804A_ABST
    Figure CN120035804A_ABST
Patent Text Reader

Abstract

A hand tracking platform generates a gesture component for use as user input into an application of an augmented reality (AR) system. In some examples, a hand tracking platform uses a camera component of an AR system to generate real-world scene environment frame data based on gestures being made by a user of the AR system. The hand tracking platform identifies a gesture component based on the real-world scene environment frame data and generates gesture component data based on the gesture component. The application utilizes the gesture component data as user input in a user interface of the application.
Need to check novelty before this filing date? Find Prior Art

Claims

1. A computer-implemented method, include: generating, by one or more processors, real-world scene environment frame data of a gesture being performed by a user of an augmented reality (AR) system using a camera component of the AR system; identifying, by the one or more processors, a gesture component of the gesture based on the real-world scene environment frame data; generating, by the one or more processors, gesture component data based on the gesture components; as well as The gesture component data is utilized, by the one or more processors, as user input in a user interface of an application of the AR system.

2. The computer-implemented method of claim 1, in, Identifying the posture components further comprises: identifying landmark features based on the real-world scene environment frame data; Generating skeleton model data according to the landmark features; and The posture components are identified based on the skeletal model data.

3. The computer-implemented method of claim 2, in, The posture component is a hand shape posture component composed of the hand shape features of the skeleton model data.

4. The computer-implemented method of claim 3, in, The hand posture component is composed of redundant features of the skeleton model data.

5. The computer-implemented method of claim 1, in, The gesture component is a rotation gesture component composed of a rotation mark of the user's hand.

6. The computer-implemented method of claim 1, in, The gesture component is an incremental motion gesture component composed of incremental motion tags that describe the amount of rotation of the hand shape.

7. The computer-implemented method of claim 1, in, The AR system includes a head-mounted device.

8. A machine, include: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: generating, by the one or more processors, real-world scene environment frame data of a gesture being performed by a user of the AR system using a camera component of the AR system; identifying, by the one or more processors, a gesture component of the gesture based on the real-world scene environment frame data; generating, by the one or more processors, gesture component data based on the gesture components; and The gesture component data is utilized, by the one or more processors, as user input in a user interface of an application of the AR system.

9. The machine according to claim 8, in, The instructions further cause the machine to perform identifying the gesture components, and further cause the machine to perform operations including: identifying landmark features based on the real-world scene environment frame data; Generating skeleton model data according to the landmark features; and The posture components are identified based on the skeletal model data.

10. The machine according to claim 9, in, The posture component is a hand shape posture component composed of hand shape features in the skeleton model data.

11. The machine according to claim 10, in, The hand posture component is composed of redundant features in the skeleton model data.

12. The machine according to claim 8, in, The gesture component is a rotation gesture component composed of a rotation mark of the user's hand.

13. The machine according to claim 8, in, The gesture component is an incremental motion gesture component composed of incremental motion tags that describe the amount of rotation of the hand shape.

14. The machine according to claim 8, in, The AR system includes a head-mounted device.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors, cause a computer to perform operations, the operations include: using a camera component of an augmented reality (AR) system to generate real-world scene environment frame data of a gesture being performed by a user of the AR system; identifying a gesture component of the gesture based on the real-world scene environment frame data; generating posture component data based on the posture components; as well as The gesture component data is utilized as user input in a user interface of an application of the AR system.

16. The non-transitory machine-readable storage medium of claim 15, in, The instructions, when executed by the one or more processors, cause the computer to perform operations of identifying the gesture components, and further cause the computer to perform operations including: identifying landmark features based on the real-world scene environment frame data; Generating skeleton model data according to the landmark features; and The posture components are identified based on the skeletal model data.

17. The non-transitory machine-readable storage medium of claim 16, in, The posture component is a hand shape posture component composed of hand shape features in the skeleton model data.

18. The non-transitory machine-readable storage medium of claim 17, in, The hand posture component is composed of redundant features in the skeleton model data.

19. The non-transitory machine-readable storage medium of claim 15, in, The gesture component is a rotation gesture component composed of a rotation mark of the user's hand.

20. The non-transitory machine-readable storage medium of claim 15, in, The gesture component is an incremental motion gesture component composed of incremental motion tags that describe the amount of rotation of the hand shape.

Citation Information

Cited By

  • Tractable body-based AR system input

    US12530086B2