One-handed zoom operation for AR / VR devices

Abstract

An augmented reality (AR) system is provided. The AR system uses a combination of gestures and a direct manipulation of virtual objects (DMVO) approach to provide user selection and modification of virtual objects of the AR experience. Users are indicated that they want to interact with a virtual object of an AR experience by moving their hands to overlap the virtual object. While keeping their hands in the overlapping position, the user makes a gesture to zoom in or out the user's viewpoint of the virtual object. To end the interaction, the user moves their hands such that their hands no longer overlap with the virtual object.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. application serial number 17 / 823,810, filed on August 31, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art

[0004] A head-mounted device may be implemented with a transparent or translucent display through which a user of the head-mounted device can view the surrounding environment. Such a device enables a user to view the surrounding environment through a transparent or translucent display, and also to see objects (e.g., virtual objects such as presentations of 2D or 3D graphic models, images, videos, text, etc.) generated for display to appear as part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". A head-mounted device may also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". In a hybrid form, a camera is used to capture a view of the surrounding environment, and then the view is displayed to the user together with the augmentation on a display that blocks the user's eyes. As used herein, unless the context indicates otherwise, the term AR refers to augmented reality, virtual reality, and any mixture of these technologies.

[0005] A user of a head mounted device can access and use computer software applications to perform various tasks or participate in entertainment activities. Performing tasks or participating in entertainment activities may require inputting various commands into the head mounted device. Therefore, it is desirable to have a mechanism for inputting commands. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or action, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Additional views of the head-mounted device in .

[0009] Figure 3is a diagrammatic representation of a machine according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0010] Figure 4A is a collaboration diagram of a hand tracking platform for an AR system according to some examples.

[0011] Figure 4B Data structures according to some examples are shown.

[0012] Figure 5 is a process flow diagram of a virtual object rotation method according to some examples.

[0013] Fig. 6A and Figure 6B A zoom out gesture and operation is shown in accordance with some embodiments.

[0014] Fig. 7A and Figure 7B Magnification gestures and operations are shown according to some examples.

[0015] Figure 8 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some examples.

[0016] Fig. 9 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.

[0017] Fig.10 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION

[0018] AR systems implemented on head-mounted devices such as glasses are limited when it comes to available user input modalities. Compared to other mobile devices such as mobile phones, it is more complicated for users of AR systems to indicate user intent and invoke actions or applications. When using a mobile phone, a user can go to the home screen and click on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easy to perform on AR systems. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it is desirable to have an input modality that can allow a wider variety of inputs that can be utilized by users to indicate their intent through user input. Computer vision-based hand tracking provides such an input modality.

[0019] An example of a hand tracking input modality that can be used with an AR system is hand tracking combined with direct manipulation of virtual objects (DMVO), in which a user interface is provided to the user, which is displayed to the user in an AR overlay with 2D or 3D rendering. The rendering has the properties of a graphical model in 2D or 3D, where virtual objects located in the model correspond to interactive elements of the user interface. In this way, the user perceives the virtual objects as objects within the overlay in the user's field of view of the real world scene environment when wearing the AR system, or perceives the virtual objects as objects within the virtual world viewed by the user when wearing the AR system. In order to enable the user to manipulate virtual objects, the AR system detects the user's hands and tracks their movement, position, and / or positioning to determine the user's interaction with the virtual objects.

[0020] Gestures that do not involve DMVO provide another hand tracking input modality suitable for use with an AR system. Gestures are made by the user moving and positioning parts of the user's body, and when the user wears the AR system, those parts of the user's body can be detected by the AR system. Detectable parts of the user's body may include parts of the user's upper torso, arms, hands, and fingers. Gesture components may include the movement of the user's arms and hands, the position of the user's arms and hands in space, and the positioning of the user holding his upper torso, arms, hands, and fingers. Gestures are useful in providing an AR experience for users because they provide a way to provide user input to the AR system during the AR experience without causing the user to shift their attention away from the AR experience. As an example, in an AR experience that is an operating manual for a piece of machinery, the user can simultaneously view the piece of machinery in a real-world scene environment through the lens of the AR system, view the AR overlay on the real-world scene environment of the machinery, and provide user input to the AR system.

[0021] By combining hand tracking with DMVO gesture methods, an improved user input modality is provided to users of the AR system. The AR system uses a combination of gestures and DMVO methods to provide users with selection and modification of virtual objects of the AR experience. Users indicate that they want to interact with the virtual object by moving their hands to overlap the virtual object of the AR experience. While keeping their hands in the overlapping position, the user can make a zoom-in gesture to zoom in on the virtual object, and a zoom-out gesture to zoom out on the virtual object. To end the interaction, the user moves their hand so that their hand no longer overlaps the virtual object.

[0022] Other technical features may be readily apparent to those skilled in the art from the accompanying drawings, descriptions and claims.

[0023] Figure 1is a head mounted AR system according to some examples (e.g., Figure 1 100). The glasses 100 may include a frame 102 made of any suitable material such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0024] Frame 102 also includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0025] The glasses 100 may include a computing system such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be at least partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include multiple processors, memories, and various communication components that share a common power supply. As discussed below, the various components of the computer 120 may include low-power circuitry, high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 902 discussed below.

[0026] The computer 120 also includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0027] The glasses 100 include a first camera or left camera 114 and a second camera or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may also include biometric sensors, position sensors, motion sensors, etc.

[0028] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real-world scene environment scene.

[0029] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 120 and the right temple piece 122. The touchpad 126 is generally arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than this. Additional user input can be provided by one or more buttons 128, which, in the example shown, are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a device by which the glasses 100 can receive input from the user of the glasses 100.

[0030] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0031] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0032] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, a lens, or a prism). The light 208 emitted by the projector 204 encounters the diffractive structure of the waveguide of the near-eye display 206, which directs the light to the right eye of the user to provide an image superimposed on or in the right optical element 110 with a view of the real-world scene environment seen by the user. Similarly, the light 214 emitted by the projector 212 encounters the diffractive structure of the waveguide of the near-eye display 216, which directs the light to the left eye of the user to provide an image superimposed on or in the left optical element 108 with a view of the real-world scene environment seen by the user. The combination of the GPU, the front optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay with the user's view of the real-world scene environment, including displaying a user interface to the user of the glasses 100.

[0033] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to a user in the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.

[0034] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, voice input, or on an associated device (e.g., Fig. 9 The user may interact with the glasses 100 based on touch input on a client device 926 shown in FIG. 1 and / or hand movement, position, and orientation detected by the glasses 100 .

[0035] In some examples, glasses 100 include a standalone AR system that provides an AR experience to a user of glasses 100. In some examples, glasses 100 are a component of an AR system that includes one or more other devices that provide additional computing resources and / or additional user input and output resources. Other devices may include smartphones, general-purpose computers, etc.

[0036] Figure 3 is a diagrammatic representation of a machine 300 in which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as an AR system (e.g., Figure 1The computer 120 of the glasses 100) can be configured to be configured to be operable by the machine 300. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methods described herein. The instructions 310 transform a general purpose, non-programmed machine 300 into a specific machine 300 that performs the functions described and illustrated in the manner described. The machine 300 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300, in combination with other components of the AR system, may be used as, but not limited to: a server, a client, a computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web (network) device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 310 specifying actions to be taken by the machine 300. In addition, although a single machine 300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 310 to perform any one or more of the methods discussed herein.

[0037] The machine 300 may include a processor 302, a memory 304, and an I / O device interface 306 that may be configured to communicate with each other via a bus 344. In an example, the processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include a processor 308 and a processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (eg, a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0038] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. The instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within a non-transitory machine-readable medium 320 within the storage unit 318, within one or more of the processors 302 (e.g., within a processor's cache memory), or within any suitable combination thereof during execution thereof by the machine 300.

[0039] I / O device interfaces 306 couple machine 300 to I / O devices 346. One or more of I / O devices 346 may be components of machine 300, or may be separate devices. I / O device interfaces 306 may include various interfaces to I / O devices 346 that are used by machine 300 to receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O device interfaces 306 included in a particular machine will depend on the type of machine. It will be appreciated that I / O device interfaces 306 may include various interfaces to I / O devices 346. Figure 3 306. In various examples, the I / O device interface 306 may include an output component interface 328 and an input component interface 332. The output component interface 328 may include an interface to a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component interface 332 may include an interface to an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the location and / or force of a touch or touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0040] In other examples, the I / O device interface 306 may include a biometric component interface 334, a motion component interface 336, an environment component interface 338, or a positioning component interface 340, as well as various other component interfaces. For example, the biometric component interface 334 includes an interface to a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component interface 336 may include an interface to an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component interface 338 may include, for example, an interface to an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that may provide indications, measurements, or signals associated with the surrounding physical environment. The positioning component interface 340 includes an interface to a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer from which the air pressure is detected and the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0041] Various technologies may be used to implement communications. I / O device interface 306 also includes a communication component interface 342 that is operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication component interface 342 may include an interface to a network interface component or other suitable device that interfaces with network 322. In other examples, communication component interface 342 may include an interface to a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a wireless communication ... Components (e.g. Low energy consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0042] In addition, the communication component interface 342 may include an interface to a component operable to detect an identifier. For example, the communication component interface 342 may include an interface to a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or an acoustic detection component (e.g., a microphone for an audio signal identifying a tag). In addition, various information may be derived via the communication component interface 342, such as a location derived via Internet Protocol (IP) geolocation, ... Location derived from signal triangulation, location derived via detection of NFC beacon signals that can indicate a specific location, etc.

[0043] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.

[0044] Instructions 310 may be sent or received via a network interface device (e.g., a network interface component included in communication component interface 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 322. Similarly, instructions 310 may be sent or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0045] Figure 4A is a collaboration diagram of a hand tracking platform 400 for an AR system (e.g., glasses 100), and Figure 4BData structures according to some examples are shown. The hand tracking platform 400 includes a computer vision SoC 428 that hosts a hand tracking input pipeline 424 for processing hand tracking input into an AR system and one or more application SoCs 414 that hosts an AR application provided to a user of the AR system, such as an AR application 412. The hand tracking platform 400 also includes a gesture component framework 430 that provides one or more application programming interfaces (APIs) that provide a communication channel between components of the hand tracking input pipeline 424 and the AR application 412. In some examples, the application SoC 414 in the one or more application SoCs serves as the core processing system of the AR system and hosts an operating system for the AR system.

[0046] The hand tracking input pipeline 424 includes a camera component 402, such as Figure 1 The camera 114 and the camera 116 capture video frame data of the real-world scene environment from the perspective of the user of the AR system, and generate tracking video frame data 418 based on the captured video frame data. The tracking video frame data 418 includes tracking video frame data of detectable parts of the user's body (including parts of the user's upper body, arms, hands, and fingers) when the user makes gestures. The tracking video frame data includes: video frame data of the movement of parts of the user's upper body, arms, and hands when the user makes gestures or moves their hands and fingers to interact with the real-world scene environment; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves their hands and fingers to interact with the real-world scene environment; and video frame data of the positioning of the user holding their upper body, arms, hands, and fingers when the user makes gestures or moves their hands and fingers to interact with the real-world scene environment; the camera component 402 transmits the tracking video frame data 418 to the skeleton model classifier 404.

[0047] The skeletal model classifier 404 identifies landmark features based on the tracking video frame data 418. The skeletal model classifier 404 generates skeletal model data 422 based on the identified landmark features. The landmark features include landmarks on parts of the user's upper body, arms, and hands in the real-world scene environment. The skeletal model data 422 includes data representing a skeletal model of a user's body part (e.g., their hands and arms). In some examples, the skeletal model data 422 also includes landmark data, such as landmark identification, location in the real-world scene environment, segments between joints, and classification information of one or more landmarks associated with the user's upper body, arms, and hands.

[0048] In some examples, the skeleton model classifier 404 extracts feature data from the tracking video frame data 418 using one or more computer vision methods, including but not limited to Harris corner detection, Shi-Tomasi corner detection, scale-invariant feature transform (SIFT), accelerated robust features (SURF), accelerated segmentation test features (FAST), oriented FAST and rotation briefing (ORB), etc. The skeleton model classifier 404 generates skeleton model data 422 of the skeleton model of the hand of the user 524 based on the extracted feature data. The skeleton model classifier 404 generates skeleton model data 422 of the skeleton model based on the extracted feature data. In some examples, the skeleton model classifier 404 generates the skeleton model data 422 using a geometric method for generating a skeleton model and one or more previously generated geometric models. In some examples, the skeleton model classifier 404 generates the skeleton model data 422 based on the skeleton model generation model generated using an artificial intelligence method and a previously generated using a machine learning method to classify the extracted feature data. In some examples, the skeleton model generation model includes but is not limited to a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, a K nearest neighbor model, etc. In some examples, the machine learning method may include but is not limited to supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, anomaly detection, etc.

[0049] In some examples, the skeletal model classifier 404 identifies joint features and generates low-level joint posture components representing the user's joints. These posture components can be virtual representations of natural joint positioning on the user's body (such as but not limited to fingertips, finger joints, wrists, elbows, shoulders, etc.). 3D markers that can be defined on the user can also be included in this category, even if it is not associated with a physical joint.

[0050] Skeletal model classifier 404 transmits skeletal model data 422 to gesture component classifier 406 . In some examples, skeletal model classifier 404 transmits skeletal model data 422 to AR application 412 according to an API of gesture component framework 430 .

[0051] Posture component classifier 406 receives skeletal model data 422 from skeletal model classifier 404 and identifies posture components based on skeletal model data 422. Posture component classifier 406 generates posture component data 420 based on the identified posture components. Posture component data 420 includes data of the identified posture components including identifications of the identified posture components.

[0052] In some examples, the posture component data 420 includes a confidence value indicating the confidence with which a particular posture component is identified in the skeletal model data 422. Although individual differences may occur, the posture component classifier 406 estimates each hand shape posture component individually to match the user's instantaneous hand configuration and provides a confidence in doing so (e.g., a 0.0 confidence match indicates that the user's hand does not match the hand shape at all, and a 1.0 confidence match indicates that the user's hand completely matches the hand shape). In some examples, the posture component classifier 406 scales the confidence values ​​so that the specified value represents a natural boundary between discrete acceptance or rejection of the hand shape matching the user's hand. For example, a threshold of 0.5 can be used as a natural boundary. An identified hand shape with a confidence value of 0.5 or greater is accepted as a matching hand shape. An identified hand shape with a confidence value less than 0.5 is not accepted as a matching hand shape. In some examples, if a more sophisticated decision is required, the posture component classifier 406 includes individual confidence values ​​in the posture component data 420. In some examples, posture component classifier 406 provides posture component data 420 to other applications so that the other applications can estimate a broader set of postures based on individual confidence values.

[0053] In some examples, the gesture component classifier 406 identifies hand shape gesture components composed of hand shape features in the skeletal model data 422, which are different configurations of the user's hand. The hand shape gesture components include finger configurations (bends, tilts, and relative positioning) for a given hand of the user. In some examples, the gesture component classifier 406 identifies a defined subset of a set of possible hand shapes, where the subset of possible hand shapes is based on a use case.

[0054] In some examples, the posture component classifier 406 identifies hand posture components that are composed of redundant features in the skeletal model data 422. In some examples, the posture component classifier 406 does not view hand posture components as disjoint categories of finger configurations, but rather as clusters of such configurations that have been accepted. This means that a defined set of hand posture components may include hand posture components that are intentionally exclusive, such that if a hand posture component is accepted / recognized, other hand postures should not be accepted / recognized. However, a defined set of hand postures may also include other redundant hand posture components, such as those that are intended to intersect or where one hand posture component is more specific than another hand posture component.

[0055] Figure 4A Table 1 432 is shown listing examples of gesture component identifications for hand shape gesture components according to some examples. These examples provide a level of detail for many use cases and can be grouped into broader categories.

[0056] For example, for a pinch gesture (using thumb and index finger), a reasonable set of hand shapes is G, G_CLOSED, G_INDEX_CURVED, G_OPEN, 0_NUM, O_RING_PINKY, O_FLAT, 9_NUM, 9_FLAT.

[0057] As another example, a hand shape group including B, B_FLAT, B_THUMBOUT, B_BENT, B_BENT_THUMBOUT, C may be used to recognize a slide gesture.

[0058] In some examples, gesture component classifier 406 identifies the best matching gesture components based on determining the gesture components that best match features of skeletal model data 422. For example, gesture component classifier 406 determines the gesture component or group that a given hand is most likely to match at a given moment.

[0059] In some examples, based on grouping gesture components and then identifying members of the group, gesture component classifier 406 identifies a gesture component when any member of the group is identified in skeletal model data 422. For example, a developer of an AR system defines groups of gesture components, and the defined groups are used as gesture components, which are the union of gesture components in the group, that is, if a gesture component in a group is identified, then the group is identified. In some examples, gesture component classifier 406 uses the definitions of gesture components and groups to determine the user's intent to make a particular finger configuration.

[0060] In some examples, the posture component classifier 406 identifies spatial posture components composed of spatial data features of the skeletal model data 422. A spatial posture component is a specific aspect of any spatial data that can be visually perceived. For example, useful reference spatial posture components are defined, which make the data more informative. The described 3D data can be transformed into these spatial posture components. These spatial posture components also provide a natural discretization option for certain data. For example, hand positioning can be discretized into natural, extended, and retracted categories relative to the user's body in space, which can be further discretized if normalized by the current arm length or shoulder width.

[0061] In some examples, the posture component classifier 406 identifies a derived continuous posture component composed of derived continuous features of the skeletal model data 422. A derived continuous feature is a feature that can be extracted at multiple timestamps and thus forms a continuous data stream. In some examples, the derived continuous feature posture component includes a specified level of smoothing.

[0062] In some examples, the posture component classifier 406 identifies a distance posture component composed of distance features of the skeletal model data 422. The distance feature is derived from the distance between two or more specified points of the user's body (such as but not limited to fingertips, palms, backs of hands, wrists (inside, outside, inner and outer edges), ends of fists, etc.). In addition, the specified points may also include parts of the user's body that are not on the hands, such as but not limited to the face, upper body, etc.

[0063] In some examples, posture component classifier 406 identifies symmetric posture components consisting of symmetry features of skeletal model data 422. Symmetry features describe complete or partial symmetries included in hand data defined continuously in a sequence of timestamps. Symmetry features extract information that is independent of positioning or motion and can be used to express a measure of how accurately the shape of one hand is a reflection of the shape of another hand.

[0064] In some examples, gesture component classifier 406 identifies motion gesture components composed of motion markers of skeletal model data 422. The motion marker is a continuous 3D trajectory determined for the hand that is optimized for the shape of the 3D trajectory. In some examples, the motion marker may have a local offset for a short period of time compared to the specified 3D trajectory model, which will decrease over time, but the overall movement of the hand will still match the 3D trajectory pattern of the motion marker in terms of geometric properties and shape.

[0065] In some examples, gesture component classifier 406 identifies a positioning gesture component consisting of positioning markers of skeletal model data 422. In contrast to motion markers, positioning markers are optimized for positioning of the user's hand. Positioning marker features are consistent and responsive to the movement of the user's hand. Since minimal latency in the detection of positioning marker features takes precedence over the positioning accuracy of the user's hand, the motion trajectory of the user's hand may cause artifacts.

[0066] In some examples, gesture component classifier 406 identifies interaction gesture components consisting of interaction markers of skeletal model data 422. Interaction markers are specific motion markers of the hand that target natural interaction points based on hand shape. For example, motion markers can include measurements of the farthest points of a user's fingers from a corresponding wrist. For example, when pointing with an index finger, the farthest point may be the tip of the index finger, if pointing with an open index and middle finger, or pointing with a flat hand, the farthest point may be the tip of the middle finger.

[0067] In some examples, the posture component classifier 406 identifies a rotational posture component consisting of a rotation marker of the skeletal model data 422. The rotation marker is similar to the positioning marker, but consists of the 3D rotation of the hand at a given time. This 3D rotation together with the positioning marker defines a rigid transformation of the hand description.

[0068] In some examples, the gesture component classifier 406 identifies an incremental motion gesture component consisting of incremental motion tags of the skeletal model data 422. The incremental motion tags describe the amount of rotation of the hand shape, positioning, and / or rotation change. In some examples, the fact that the hand shape or configuration has changed but not a specific change is sufficient to be recognized. For example, at the end of a gesture that has been held for a period of time, it is followed by another gesture that represents the release of the held gesture.

[0069] In some examples, gesture component classifier 406 identifies a pinch gesture component comprised of the tightness of pinch markers of skeletal model data 422. The tightness of pinch markers is a continuous assessment of how closed a pinch or gripper position is.

[0070] In some examples, the posture component classifier 406 identifies time segment posture components based on the time segmentation of the skeleton model data 422. The time segments vary from posture to posture. The data used to determine the time segments are continuous so as to capture temporal features. In some examples, for manual gestures, the selection of time segments is based on the general movement of the hand. For example, the posture component classifier 406 detects the local extrema of the curvature of the movement of the hand and uses a sequence of two local extrema to determine the segment boundary and the segment interval between the two local extrema.

[0071] In some examples, posture component classifier 406 identifies aggregated posture components of skeletal model data 422 based on aggregating multiple posture components across multiple time segment boundaries. In this way, simple positioning continuous features can be aggregated to identify positioning across one or more time segment boundaries and similarly within one or more time segment intervals.

[0072] In some examples, gesture component classifier 406 identifies continuous motion gesture components consisting of continuous motion time segments of skeletal model data 422. Continuous motion time segments are time segments with explicit motion gesture components and their derivatives identified as additional features such as hand displacement or hand velocity.

[0073] A pause or no motion is a time segment of continuous motion. A pause includes a time segment where the user's hand has little to no motion. A pause may also indicate a hold. In some examples, a pause has no additional characteristics other than its duration.

[0074] A simple motion is a continuous temporal segment of motion where the specific production of the motion (e.g., hand positioning) is irrelevant to the motion, only the displacement and duration are relevant. For example, a large sweeping motion with the arm where hand positioning is unimportant is a simple motion. Additional features may include, but are not limited to, displacement vector, average velocity, and peak velocity.

[0075] Arc motion is a continuous time segment of motion that has a measurable arc in a predetermined gesture motion. For example, stepping motion includes a measurable and unique vertical motion. Additional features of arc motion may include additional features of simple motion, and may also include the direction and magnitude of the arc.

[0076] A well-defined start or stop motion is a continuous time segment of motion with an abrupt start or stop. The distinguishing feature of a well-defined start or stop is a start or stop motion with an abrupt start or end with uneven acceleration. For example, pointing at something with a well-defined stop has a motion that starts out arbitrary and fuzzy but ends sharply. As another example, a flick gesture is an example of an opposite (start) motion, where the start is definite and the end is indefinite.

[0077] Complex motion is a continuous motion time segment spanning multiple time segments with significant consistency, such as but not limited to repetitive (or back and forth) motion, jitter, etc. In some examples, complex motion may also include additional features, such as but not limited to the number of repetitions, amplitude, repetition frequency, etc.

[0078] In some examples, the posture component classifier 406 uses a geometric method to compare one or more skeletal models included in the skeletal model data 422 with previously generated posture component models, and generates posture component data 420 including the identified posture components based on the comparison.

[0079] In some examples, the posture component classifier 406 uses artificial intelligence methods and posture component models previously generated using machine learning methods to identify posture components based on the skeleton model data 422. In some examples, the posture component models include but are not limited to neural networks, learning vector quantization networks, logistic regression models, support vector machines, random decision forests, naive Bayes models, linear discriminant analysis models, and K-nearest neighbor models. In some examples, machine learning methods include but are not limited to supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0080] Gesture component classifier 406 transmits gesture component data 420 to gesture classifier 408 and gesture text input classifier 410 . In some examples, gesture component classifier 406 transmits gesture component data 420 to AR application 412 using an API of gesture component framework 430 .

[0081] The gesture classifier 408 receives the gesture component data 420 and recognizes a gesture based on the gesture component data 420. The gesture classifier 408 generates gesture input event data 426 based on the recognized gesture. In some examples, the gesture classifier 408 recognizes the gesture based on a comparison of the gesture components recognized in the gesture component data 420 with a gesture recognition model that recognizes a specific gesture. For example, referring to Table 1, for a gesture of enlarging a virtual object or increasing the size of a virtual object, possible hand gesture components include R_180 434 (hand rotated 180 degrees or palm facing the user), 3_CLOSED 440 (hand clenched into a fist), 1_VELOCITY 438 (hand slowly transitions from one gesture to another), G_OPEN 442 (hand open), and 1_HORIZ 436 (hand parallel to the floor). In another example, for a gesture of reducing the size of a virtual object or shrinking a virtual object, the hand gesture components include G_OPEN 442 (hand open), 1_HORIZ 436 (hand parallel to the floor), 1_VELOCITY 438 (hand slowly transitions from one pose to another), 3_CLOSED 440 (hand clenched into a fist), and R_180 434 (hand rotated 180 degrees or palm facing the user).

[0082] In some examples, posture classifier 408 uses artificial intelligence methods and one or more posture models previously generated using machine learning methods to recognize postures based on posture component data 420. In some examples, posture models include, but are not limited to, neural networks, learning vector quantization networks, logistic regression models, support vector machines, random decision forests, naive Bayes models, linear discriminant analysis models, and K-nearest neighbor models. In some examples, machine learning methods include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0083] In some examples, gesture classifier 408 recognizes a gesture based on parsing gesture component data 420 using a previously determined gesture grammar.

[0084] The gesture text input classifier 410 also receives gesture component data 420 from the gesture component classifier 406. The gesture text input classifier 410 recognizes symbols based on the gesture component data 420. The gesture text input classifier 410 generates symbol input event data 416 based on the recognized symbols. In some examples, the gesture classifier 408 recognizes symbols based on comparison of gesture components in the gesture component data 420 with symbol models that identify specific characters, words, and commands.

[0085] In some examples, the gesture text input classifier 410 uses artificial intelligence methods and one or more symbol models previously generated using machine learning methods to recognize symbols based on the gesture component data 420. In some examples, the symbol models include, but are not limited to, neural networks, learning vector quantization networks, logistic regression models, support vector machines, random decision forests, naive Bayes models, linear discriminant analysis models, and K-nearest neighbor models. In some examples, machine learning methods include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0086] In some examples, the gesture text input classifier 410 recognizes symbols based on parsing the gesture component data 420 using a previously determined symbol grammar.

[0087] In some examples, gesture text input classifier 410 transmits symbol input event data 416 to AR application 412 using an API of gesture component framework 430 .

[0088] An AR application (e.g., AR application 412) executed by an AR system is a consumer of data generated by hand tracking input pipeline 424. The AR system executes AR application 412 to provide a user interface, such as an AR experience, to a user of the AR system using skeletal model data 422, gesture component data 420, gesture input event data 426, and symbol input event data 416 as input modalities, according to the purpose of the AR application 412.

[0089] In some examples, components of the AR system hosted by computer vision SoC 428 (e.g., camera component 402 and skeletal model classifier 404) communicate using a shared memory buffer. In some examples, skeletal model classifier 404 publishes skeletal model data 422 on a shared memory buffer that can be accessed by components outside of hand tracking input pipeline 424 and hosted by application SoC 414 (e.g., AR application 412).

[0090] In some examples, components of the AR system hosted by computer vision SoC 428 (e.g., gesture component classifier 406, gesture classifier 408, and gesture text input classifier 410) respectively transmit data (e.g., gesture component data 420, gesture input event data 426, and symbol input event data 416) to components of the AR system hosted by application SoC 414 using IPC methods within computer vision SoC 428.

[0091] In some examples, components of the AR system hosted by application SoC 414 communicate data with components hosted by computer vision SoC 428 using IPC method calls.

[0092] In some examples, the hand tracking input pipeline 424 continuously generates and publishes symbolic input event data 416, gesture input event data 426, skeletal model data 422, and skeletal model data 422 based on tracking video frame data 418 generated by one or more cameras of the AR system.

[0093] In some examples, any of the camera component 402, skeletal model classifier 404, posture component classifier 406, posture classifier 408, and / or posture text input classifier 410 can use lazy evaluation in which a given posture component is evaluated only on demand and only registered events (and their requirements) are calculated.

[0094] In some examples, the hand tracking platform 400 uses discretized and / or higher-level features and events, such as discrete orientations or high-level hand shape events for gesture components, thereby avoiding any user-specific features that could be used for user identification. Similarly, features like abstracted and derived motion markers can provide fine-grained input data, similar to mouse motion, when a minimal amount of data is shared. If communication is limited to an appropriate subset of gesture components that corresponds to the needs of the application, biometric data (such as hand or finger size) cannot be derived.

[0095] In some examples, the posture component classifier 406 identifies the posture components directly based on the tracking video frame data 418 without taking the intermediate step of generating the skeleton model data 422 using the skeleton model classifier 404. In some examples, the posture component classifier 406 uses artificial intelligence methods and posture component models previously generated using machine learning methods to identify posture components based on the tracking video frame data 418. In some examples, the posture component models include but are not limited to neural networks, learning vector quantization networks, logistic regression models, support vector machines, random decision forests, naive Bayes models, linear discriminant analysis models, and K-nearest neighbor models. In some examples, machine learning methods include but are not limited to supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0096] Figure 5 is a process flow chart of a virtual object interaction method according to some examples. Fig. 6A and Figure 6B is an illustration of user interaction with a virtual object during a reduction or shrinking of the size of the virtual object, according to some examples. Fig. 7A and Figure 7B is an illustration of user interaction with a virtual object during the increase or enlargement of the size of the virtual object according to some examples. A virtual object interaction method 500 is provided by an AR system (e.g., Figure 1The glasses 100 are used to provide a continuous real-time input modality to a user of the AR system in which the user interacts with a virtual object (e.g., virtual object 522) using gestures. The virtual object can be an integral part of the AR experience provided to the user by the AR system using an AR application. The AR application can be a useful application, such as a maintenance guide, an interactive map, an interactive tour guide, a tutorial, etc. The AR application can also be an entertainment application, such as a video game, an interactive video, etc.

[0097] In operation 502, the AR system displays a virtual object, such as virtual object 522, in an AR experience provided to the user by the AR system. For example, the AR system uses one or more cameras (e.g., ( Figure 1 The AR system uses a left camera 114 and a right camera 116 to capture video frame data of a real-world scene being viewed by a user of the AR system. At the same time, when capturing real-world scene video frame data, the AR system uses ( Fig. 9 The tracking component 940 captures tracking data for the AR system. The tracking data includes the orientation and position data of the AR system within the real world scene. The AR system generates a 3D model in the 3D coordinate system of the real world scene based on the real world scene video frame data and the tracking data. The AR system creates a virtual object 522 in the 3D model, such that the virtual object 522 includes 3D geometric data representing the virtual object 522 in the 3D coordinate system and graphic data defining how the virtual object 522 will be displayed to the user. The AR system uses Figure 1 The optical engine described in the embodiment of the present invention is used to provide an AR experience to a user of the AR system, wherein the AR experience includes a display of a 3D rendering of a virtual object 522, which appears to the user as if the virtual object is part of a real-world scene.

[0098] In operation 504, the AR system receives the skeleton model data 422 from the hand tracking input pipeline 424 of the AR system, and detects the position of the hand of the user 524 based on the skeleton model data 424. For example, the hand tracking input pipeline 424 uses a camera having one or more cameras (e.g., Figure 1 The camera assembly 402 of the left camera 114 and the right camera 116 captures tracking video frame data 418 of the user 524's hand as the user views and interacts with the virtual object 522. The virtual object 522 is provided to the user by the AR system as part of the AR experience. Figure 4A and Figure 4B More fully described, the skeletal model classifier 404 of the hand tracking input pipeline 424 receives the tracking video frame data 418 from the camera device component 402 and generates a skeletal model of the hand of the user 524 based on the tracking video frame information 418, such as (respectively Fig. 6A and Figure 6B ) skeleton model 602a and skeleton model 602b and (respectively Fig. 7A and Figure 7B The hand tracking input pipeline 424 provides the data of the skeleton model as part of the skeleton model data 422 to the AR system.

[0099] Each skeleton model consists of one or more links (e.g. Fig. 6A One or more nodes linked together by a link 606), such as ( Fig. 6A The one or more nodes include 3D coordinate data of a specified feature of the hand of the user 524 in the 3D model of the real world scene. The AR system uses the 3D coordinate data of one or more nodes of the skeletal model to determine the position of the hand of the user 524 within the 3D model and within the real world scene.

[0100] In operation 506, the AR system determines whether the virtual object 522 and the hand of the user 524 overlap in the 3D model of the real world scene based on the skeleton model data 422 and the 3D coordinate data of the virtual object 522. For example, the AR system determines whether the skeleton model and the virtual object 522 intersect with each other in the 3D model using the 3D coordinate data of one or more nodes of the skeleton model and the 3D coordinate data of the virtual object 522. The intersection indicates that the user's hand overlaps with the virtual object 522, and the absence of the intersection indicates that the user's hand does not overlap with the virtual object 522.

[0101] In some examples, the overlap of the hand of user 524 with virtual object 522 is determined based on the user's viewpoint, where the coordinates on the horizontal axis of the 3D coordinate system and the coordinates on the vertical axis of the 3D coordinate system are used to determine that the hand of user 526 overlaps with virtual object 522. For example, when the overlap is determined from the user's perspective, the coordinates on the depth axis of the 3D coordinate system are ignored.

[0102] like Fig. 6A As shown, the AR system determines that the hand of the user 524 overlaps with the virtual object 522, as shown by at least the index finger node 604 overlapping with the virtual object 522. Fig. 7A As shown, the AR system determines that the hand of the user 524 overlaps with the virtual object 522 , as shown by at least the middle finger joint node 704 overlapping with the virtual object 522 .

[0103] In some examples, the AR system uses one or more computer vision methods to determine whether the hand of the user 524 overlaps with the virtual object 522 based on the tracking video frame data 418, and the computer vision methods include but are not limited to Harris corner detection, Shi-Tomasi corner detection, scale-invariant feature transform (SIFT), accelerated robust features (SURF), accelerated segmentation test features (FAST), oriented FAST, and rotation briefing (ORB). The AR system extracts feature data of the hand of the user 524 and determines the 3D coordinate data of the hand of the user 524 based on the feature data. The AR system determines the intersection of one or more features of the hand of the user 524 based on the 3D coordinate data of the hand of the user 524 and the 3D coordinates of the virtual object.

[0104] In operation 506, if the AR system determines that the virtual object 522 does not overlap with the hand of the user 524, the AR system transitions back to operation 504 and continues to determine the position of the hand of the user 524 until an overlap is detected between the hand of the user 524 and the virtual object 522. If the AR system determines that the hand of the user 524 overlaps with the virtual object 522, the AR system transitions to operation 508.

[0105] In operation 508, the AR system receives gesture input event data 426 from the hand tracking input pipeline 424 and determines the gesture that the user is currently performing. For example, when the user views and interacts with the virtual object 522, the AR system uses the camera component 402 to capture tracking video frame data 418 of the user's 524 hand. Figure 4A and Figure 4B As described more fully, the hand tracking input pipeline 424 generates gesture input event data 426 based on the tracking video frame data 418. In some examples, such as Fig. 6A and Figure 6B As shown, the gesture identified by the gesture classifier 408 of the hand tracking input pipeline 424 is a reduction or zoom-out gesture being made by the user while interacting with the virtual object 522. Figure 4B, such a gesture may be composed of gesture components, including G_OPEN 442 (hand open), 1_HORIZ 436 (hand parallel to the floor), 1_VELOCITY 438 (hand slowly transitions from one gesture to another), 3_CLOSED 440 (hand clenched into a fist), and R_180 434 (hand rotated 180 degrees or palm facing the user). In another example, a size increase or zoom in gesture includes hand gesture components, including R_180 434 (hand rotated 180 degrees or palm facing the user), 3_CLOSED 440 (hand clenched into a fist), 1_VELOCITY 438 (hand slowly transitions from one gesture to another), G_OPEN 442 (hand open), and 1_HORIZ 436 (hand parallel to the floor).

[0106] In operation 510, the AR system determines whether the gesture included in the gesture input event data 426 corresponds to an operation to be applied to the virtual object 522. For example, referring to Fig. 6A and Figure 6B , the size reduction or zoom out gesture corresponds to an operation of reducing the size of the virtual object 522, so that it appears that the user's viewpoint is zooming out. Fig. 6A 5 , the user 524's hand overlaps the virtual object 522, and the user is opening their hand parallel to the floor and slowly closing their hand, transitioning from an open position to a closed position with their hand rotated 180 degrees or with the palm facing the user, as shown in FIG. Figure 6B shown.

[0107] In some examples, reference Fig. 7A and Figure 7B , the size increase or zoom in gesture corresponds to an operation of increasing the size of the virtual object 522 so that it appears that the user's viewpoint is zooming in. Fig. 7A In FIG. 5 , the hand of user 524 overlaps with virtual object 522, and the user is holding a clenched fist and rotating their hand 180 degrees or with the palm facing the user. Figure 7B As shown, the user slowly opens their hands, transitioning from a closed hand position to an open hand position with their hands parallel to the floor.

[0108] If, in operation 510, the AR system determines that the gesture does not correspond to a specific operation to be performed on the virtual object, the AR system transitions to operation 516 and continues as described with reference to operation 516. If, in operation 510, the AR system determines that the gesture corresponds to a specific operation to be performed on the virtual object, the AR system transitions to operation 512.

[0109] In operation 512, the AR system performs a specific operation on the virtual object based on the gesture being made by the user. In some examples, in response to detecting Fig. 6A and Figure 6B In some examples, in response to detecting a decrease in size or zoom out gesture, the AR system performs an operation to reduce the size of virtual object 522 so that it appears to the user that their viewpoint is shrinking. Fig. 7A and Figure 7B With the size increase or zoom in gesture, the AR system performs an operation of increasing the size of the virtual object 522 so that it appears to the user that their viewpoint is zooming in on the virtual object 522.

[0110] In operation 514, the AR system redisplays the virtual object 522 based on the operation performed on the virtual object 522. For example, if the AR system has performed an operation to increase the size of the virtual object 522, Figure 7B As shown in FIG. 5 , the AR system redisplays the virtual object 522 in an increased size. If the AR system has performed an operation of reducing the size of the virtual object 522, then Figure 6B As shown, the AR system redisplays the virtual object 522 at a reduced size.

[0111] In operation 516 , the AR system determines the location of the hand of the user 524 , as described more fully with reference to operation 504 .

[0112] In operation 518, the AR system determines whether the hand of the user 524 still overlaps with the virtual object 522, as described more fully with reference to operation 506. Based on determining that the hand of the user 524 still overlaps with the virtual object 522, the AR system transitions to operation 508 to continue processing the user's interaction with the virtual object 522. Based on determining that the hand of the user 524 no longer overlaps with the virtual object 522, the AR system ends the user's interaction with the virtual object 522 by transitioning to operation 520.

[0113] In some examples, the AR system performs the functionality of the hand tracking input pipeline 424 by leveraging various APIs and system libraries.

[0114] Figure 8 800 is a block diagram illustrating a software architecture 804 that may be installed on any one or more of the devices described herein. The software architecture 804 is supported by hardware such as a machine 802 including a processor 820, a memory 826, and an I / O component interface 838. In this example, the software architecture 804 may be conceptualized as a stack of layers in which each layer provides a specific function. The software architecture 804 includes layers such as an operating system 812, a library 808, a framework 810, and an application 806. In operation, the application 806 invokes an API call 850 through the software stack and receives a message 852 in response to the API call 850.

[0115] The operating system 812 manages hardware resources and provides public services. The operating system 812 includes, for example, a kernel 814, services 816, and drivers 822. The kernel 814 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 814 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 816 can provide other public services to other software layers. Drivers 822 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 822 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0116] The library 808 provides a low-level common infrastructure used by the application 806. The library 808 may include a system library 818 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 808 may include an API library 824, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 808 may also include various other libraries 828 to provide many other APIs to the application 806 .

[0117] The framework 810 provides a high-level common infrastructure used by the applications 806. For example, the framework 810 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 810 can provide a wide range of other APIs that can be used by the applications 806, some of which may be specific to a particular operating system or platform.

[0118] In an example, applications 806 may include a home application 836, a contacts application 830, a browser application 832, a book reader application 834, a location application 842, a media application 844, a messaging application 846, a game application 848, and a broad category of other applications such as third-party applications 840. Applications 806 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 806 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 840 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the Phone or another mobile operating system. In this example, third-party applications 840 can activate API calls 850 provided by the operating system 812 to facilitate the functions described herein.

[0119] Fig. 9 is a block diagram showing a networked system 900 including details of the glasses 100 according to some examples. The networked system 900 includes the glasses 100, a client device 926, and a server system 932. The client device 926 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 936 and / or a high-speed wireless connection 934. The client device 926 is connected to the server system 932 via a network 930. The network 930 may include any combination of wired and wireless connections. The server system 932 may be one or more computing devices that are part of a service or network computing system. The client device 926 and any elements of the server system 932 and the network 930 may be connected using a wireless network, such as a wireless network or a wireless network. Figure 8 and Figure 3 The details of the software architecture 804 or machine 300 described in are implemented.

[0120] The glasses 100 include a data processor 902, a display 910, one or more cameras 908, and additional input / output elements 916. The input / output elements 916 may include microphones, audio speakers, biometric sensors, additional sensors, or additional display elements integrated with the data processor 902. Figure 8 and Figure 3Examples of input / output elements 916 are further discussed. For example, the input / output elements 916 may include any I / O device interface 306, including the output component interface 328, the motion component interface 336, etc. Figure 2 Examples of display 910 are discussed in . In the particular examples described herein, display 910 includes displays for the left and right eyes of a user.

[0121] Data processor 902 includes image processor 906 (eg, video processor), GPU and display driver 938, tracking component 940, interface 912, low power circuitry 904, and high speed circuitry 920. The components of data processor 902 are interconnected by bus 942.

[0122] The interface 912 refers to any source of user commands provided to the data processor 902. In one or more examples, the interface 912 is a physical button that, when pressed, sends a user input signal from the interface 912 to the low-power processor 914. The low-power processor 914 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 914 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for an extended period of time can capture a still image. In some examples, the interface 912 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from the camera 908. In other examples, the interface 912 can have a software component, or can be associated with a command received wirelessly from another source such as from the client device 926.

[0123] Image processor 906 includes circuitry for receiving signals from camera 908 and processing those signals from camera 908 into a format suitable for storage in memory 924 or for transmission to client device 926. In one or more examples, image processor 906 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 908, and volatile memory used by the microprocessor in operation.

[0124] The low power circuit system 904 includes a low power processor 914 and a low power wireless circuit system 918. These elements of the low power circuit system 904 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 914 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 914 can accept user input signals from the interface 912. The low power processor 914 can also be configured to receive input signals or command communications from a client device 926 via a low power wireless connection 936. The low power wireless circuit system 918 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 918. In other examples, other low power communication systems may be used.

[0125] High-speed circuit system 920 includes a high-speed processor 922, a memory 924, and a high-speed wireless circuit system 928. High-speed processor 922 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 902. High-speed processor 922 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 934 using high-speed wireless circuit system 928. In some examples, high-speed processor 922 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 8 The high-speed processor 922, which executes the software architecture of the data processor 902, manages data transmission with the high-speed wireless circuit system 928, in addition to any other duties. In some examples, the high-speed wireless circuit system 928 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 928 can implement other high-speed communication standards.

[0126] The memory 924 includes any storage device capable of storing camera data generated by the camera 908 and the image processor 906. Although the memory 924 is shown as being integrated with the high-speed circuitry 920, in other examples, the memory 924 may be a separate, independent element of the data processor 902. In some such examples, electrical wiring may provide a connection from the image processor 906 or the low-power processor 914 to the memory 924 through a chip including the high-speed processor 922. In other examples, the high-speed processor 922 may manage the addressing of the memory 924 so that the low-power processor 914 will initiate the high-speed processor 922 whenever a read or write operation involving the memory 924 is desired.

[0127] The tracking component 940 estimates the pose of the glasses 100. For example, the tracking component 940 uses image data and associated inertial data from the camera device 908 and the positioning component interface 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking component 940 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative positioning and orientation relative to physical objects in the real-world scene environment. The tracking component 940 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 910.

[0128] The GPU and display driver 938 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 910 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 938 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world scene environment.

[0129] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 926 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 806, such as the messaging application 846.

[0130] Fig.10is a block diagram illustrating an example messaging system 1000 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1000 includes multiple instances of a client device 926 that host several applications including a messaging client 1002 and other applications 1004. The messaging client 1002 is communicatively coupled to other instances of the messaging client 1002 (e.g., hosted on respective other client devices 926), a messaging server system 1006, and a third-party server 1008 via a network 930 (e.g., the Internet). The messaging client 1002 may also communicate with the locally hosted application 1004 using an application program interface (API).

[0131] The messaging clients 1002 are able to communicate and exchange data with other messaging clients 1002 and with a messaging server system 1006 via the network 930. The data exchanged between the messaging clients 1002 and between the messaging clients 1002 and the messaging server system 1006 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0132] The messaging server system 1006 provides server-side functionality to a particular messaging client 1002 via the network 930. Although some functionality of the messaging system 1000 is described herein as being performed by the messaging client 1002 or by the messaging server system 1006, it may be a design choice to locate some functionality within the messaging client 1002 or within the messaging server system 1006. For example, it may be technically preferable to initially deploy some technologies and functionality within the messaging server system 1006, but then migrate the technologies and functionality to the messaging client 1002 where the client device 926 has sufficient processing power.

[0133] The messaging server system 1006 supports various services and operations provided to the messaging clients 1002. Such operations include sending data to the messaging clients 1002, receiving data from the messaging clients 1002, and processing data generated by the messaging clients 1002. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1000 is activated and controlled by functions available via the user interface (UI) of the messaging client 1002.

[0134] Turning now specifically to the messaging server system 1006, an application program interface (API) server 1010 is coupled to and provides a programming interface to an application server 1014. The application server 1014 is communicatively coupled to a database server 1016, which facilitates access to a database 1020 that stores data associated with messages processed by the application server 1014. Similarly, a web server 1024 is coupled to and provides a web-based interface to the application server 1014. To this end, the web server 1024 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0135] The application program interface (API) server 1010 receives and sends message data (e.g., commands and message payloads) between the client device 926 and the application server 1014. Specifically, the application program interface (API) server 1010 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1002 to activate the functions of the application server 1014. The application program interface (API) server 1010 exposes various functions supported by the application server 1014, including: account registration; login functionality; sending messages from a particular messaging client 1002 to another messaging client 1002 via the application server 1014, sending media files (e.g., images or videos) from the messaging client 1002 to the messaging server 1012, and possible access by another messaging client 1002; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 926; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1002).

[0136] The application server 1014 hosts several server applications and subsystems, including, for example, a messaging server 1012, an image processing server 1018, and a social network server 1022. The messaging server 1012 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1002. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1002. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1012.

[0137] The application server 1014 also includes an image processing server 1018 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1012.

[0138] The social network server 1022 supports various social networking functions and services and makes these functions and services available to the messaging server 1012. To this end, the social network server 1022 maintains and accesses an entity graph within the database 1020. Examples of functions and services supported by the social network server 1022 include identifying other users in the messaging system 1000 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0139] The messaging client 1002 may notify the user of the client device 926 or other users associated with such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1002 may provide notifications related to current or recent use of a game by one or more members of a user group to participants in a conversation (e.g., a chat session) in the messaging client 1002. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0140] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0141] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0142] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0143] "Machine-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0144] "Machine storage media" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media."

[0145] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0146] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0147] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method comprising: displaying, by one or more processors, a virtual object in an augmented reality (AR) experience provided to a user by the AR system; determining, by the one or more processors, an overlap between the user's hand and the virtual object using one or more cameras of the AR system; determining, by the one or more processors using the one or more cameras, a gesture being performed by the user; performing, by the one or more processors, an operation on the virtual object based on the gesture; as well as The virtual object is redisplayed, by the one or more processors, based on the operation performed on the virtual object.

2. The computer-implemented method of claim 1 , further comprising: determining, by the one or more processors, that the user's hand no longer overlaps the virtual object; as well as Based on determining that the user's hand no longer overlaps the virtual object, the one or more processors end the user's interaction with the virtual object.

3. The computer-implemented method of claim 1 , wherein: The operation of determining the overlap between the user's hand and the virtual object further includes: capturing tracking video frame data of the user's hands using the one or more cameras; generating a skeletal model of the user's hand based on the tracking video frame data; and An overlap between the user's hand and the virtual object is determined based on the skeleton model and the 3D coordinate data of the virtual object.

4. The computer-implemented method of claim 1 , wherein: The operation of determining the overlap between the user's hand and the virtual object further includes: capturing tracking video frame data of the user's hands using the one or more cameras; and An overlap between the user's hand and the virtual object is determined based on the tracking video frame data using one or more computer vision methods.

5. The computer-implemented method of claim 1 , wherein: The gesture is a zoom-in gesture, and the operation performed on the virtual object is increasing the size of the virtual object.

6. The computer-implemented method of claim 1 , wherein: The gesture is a zoom-out gesture, and the operation performed on the virtual object is reducing the size of the virtual object.

7. The computer-implemented method of claim 1 , wherein: The AR system includes a head-mounted device.

8. A computing device, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: Displaying the virtual object in an augmented reality (AR) experience provided to the user by the AR system; determining, using one or more cameras of the AR system, an overlap between the user's hand and the virtual object; determining, using the one or more cameras, a gesture being performed by the user; performing an operation on the virtual object based on the gesture; and The virtual object is redisplayed based on the operation performed on the virtual object.

9. The computing device according to claim 8, wherein: The operations also include: Determining that the user's hand no longer overlaps the virtual object; and Based on determining that the user's hand no longer overlaps the virtual object, the interaction between the user and the virtual object is ended.

10. The computing device according to claim 8, wherein: The instructions also include: capturing tracking video frame data of the user's hands using the one or more cameras; generating a skeletal model of the user's hand based on the tracking video frame data; and An overlap between the user's hand and the virtual object is determined based on the skeleton model and the 3D coordinate data of the virtual object.

11. The computing device according to claim 8, wherein: The operation of determining the overlap between the user's hand and the virtual object further includes: capturing tracking video frame data of the user's hands using the one or more cameras; and An overlap between the user's hand and the virtual object is determined based on the tracking video frame data using one or more computer vision methods.

12. The computing device according to claim 8, wherein: The gesture is a zoom-in gesture, and the operation performed on the virtual object is increasing the size of the virtual object.

13. The computing device of claim 8, wherein: The gesture is a zoom-out gesture, and the operation performed on the virtual object is reducing the size of the virtual object.

14. The computing device of claim 8, wherein: The AR system includes a head-mounted device.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: Displaying the virtual object in an augmented reality (AR) experience provided to the user by the AR system; determining, using one or more cameras of the AR system, an overlap between the user's hand and the virtual object; determining, using the one or more cameras, a gesture being performed by the user; performing an operation on the virtual object based on the gesture; as well as The virtual object is redisplayed based on the operation performed on the virtual object.

16. The non-transitory computer readable storage medium of claim 15, wherein: The operations also include: Determining that the user's hand no longer overlaps the virtual object; and Based on determining that the user's hand no longer overlaps the virtual object, the interaction between the user and the virtual object is ended.

17. The non-transitory computer-readable storage medium of claim 15, wherein: The operation of determining the overlap between the user's hand and the virtual object further includes: capturing tracking video frame data of the user's hands using the one or more cameras; generating a skeletal model of the user's hand based on the tracking video frame data; and An overlap between the user's hand and the virtual object is determined based on the skeleton model and the 3D coordinate data of the virtual object.

18. The non-transitory computer readable storage medium of claim 15, wherein: The gesture is a zoom-in gesture, and the operation performed on the virtual object is increasing the size of the virtual object.

19. The non-transitory computer-readable storage medium of claim 15, wherein: The gesture is a zoom-out gesture, and the operation performed on the virtual object is reducing the size of the virtual object.

20. The non-transitory computer readable storage medium of claim 15, wherein: The AR system includes a head-mounted device.

Images