Augmented reality visual or auditory feedback

By displaying virtual objects on the head-mounted XR device and tracking user hand movements, providing visual and auditory feedback, the problem of insufficient user interaction feedback in the prior art is solved, and the interactivity and user experience of the virtual reality experience is improved.

CN118891601APending Publication Date: 2024-11-01SNAP INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027592.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-16
Filing Date
2023-03-14
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

When existing head-mounted XR devices provide augmented reality and virtual reality experiences, they lack effective user interaction feedback mechanisms, especially in real-world environments, which make it difficult for users to interact with virtual objects easily and lack tactile feedback, affecting users. Experience and interactivity.

Method used

By displaying a virtual object on a display device and tracking the user's hand movement in combination with the camera device, detecting hand proximity and gestures, visual and auditory feedback is provided to indicate the user's interaction state with the virtual object, such as through color changes, sound Feedback and other methods to enhance the interactive experience between users and virtual objects.

Benefits of technology

Improves the interactivity between users and virtual objects, provides a more intuitive feedback mechanism, enhances the user experience, reduces dependence on real-world haptic feedback, and improves the functionality and accessibility of the XR system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118891601A_ABST
    Figure CN118891601A_ABST
Patent Text Reader

Abstract

Feedback is provided to a user of an augmented reality device or virtual reality device based on detecting a proximity of a user's hand to an interactive virtual object being displayed by the augmented reality device or virtual reality device. More specifically, a method of providing feedback in a display apparatus including an image pickup apparatus includes displaying a virtual object on the display apparatus, tracking a user's hand in a field of view of the image pickup apparatus, determining a proximity of the user's hand to the virtual object, and displaying the virtual object on the display apparatus. And provide auditory or visual user feedback to a user of the device based on the proximity of the user's hand to the virtual object.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] priority

[0002] This application claims the benefit of priority to U.S. patent application serial number 17 / 655,125, filed on March 16, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to display devices, and more particularly, to display devices for augmented reality and virtual reality. Background Art

[0004] The head-mounted device can be implemented as a transparent or translucent display, through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and also to see objects generated for display that appear as part of the surrounding environment and / or superimposed on the surrounding environment (for example, virtual objects such as 3D renderings, images, videos, text, etc.). This is generally referred to as "augmented reality" or "AR". The head-mounted device can also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". AR and VR are collectively referred to as "XR", where "X" is understood to represent "augmented" or "virtual". "As used herein, unless the context indicates otherwise, the term XR refers to one or both of augmented reality and virtual reality as traditionally understood.

[0005] Users of the head mounted device can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with the 3D user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Another view of the headset.

[0009] Figure 3 It is shown that according to some examples, Figure 1 A block diagram of a networking system 300 showing details of a head mounted device.

[0010] Figure 4A 3D user interface according to some examples is shown.

[0011] Figure 5A and Figure 5B Two examples of interactive virtual objects and their associated rules and properties are shown.

[0012] Figure 6 Shown is a UI architecture 600 that may be used to implement feedback according to some examples.

[0013] Fig. 7A and Figure 7B User feedback provided in an AR wearable device such as glasses 100 according to some examples is shown.

[0014] Figure 8 is a flowchart 800 illustrating a method of providing feedback according to some examples.

[0015] Fig. 9 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0016] Fig.10 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to some examples.

[0017] Fig.11 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0018] Some head-mounted XR devices, such as AR glasses, include a transparent or translucent display, which enables the user to look through the transparent or translucent display to view the surrounding environment. Additional information or objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) are displayed on the display and appear as part of the surrounding environment and / or superimposed on the surrounding environment to provide an augmented reality (AR) experience for the user. The display may, for example, include a waveguide that receives a light beam from a projector, but any suitable display for presenting augmented content or virtual content to the wearer may be used.

[0019] As referred to herein, the phrase "augmented reality experience" includes or refers to various image processing operations corresponding to image modification, filtering, media overlay, transformation, etc., as further described herein. In some examples, these image processing operations provide an interactive experience of a real-world environment, in which objects, surfaces, backgrounds, lighting, etc. in the real world are enhanced by computer-generated perceptual information. In this context, an "augmented reality effect" includes a collection of data, parameters, and other assets for applying a selected augmented reality experience to an image or video feed. In some examples, augmented reality effects are provided by Snap Inc. under the registered trademark LENSES.

[0020] In some examples, a 3D user interface is used to enable user interaction with a software application executing on an XR device. The 3D user interface includes virtual objects displayed to the user by the XR device in a 3D rendering displayed to the user. In the case of AR, the user perceives the virtual objects as objects in the real world viewed by the user while wearing the XR device. In the case of VR, the user perceives the virtual objects as objects in the virtual world viewed by the user while wearing the XR device. To enable the user to interact with the virtual objects, the XR device detects the positioning and movement of the user's hands, and uses these positioning and movement of the hands to determine the user's intent in manipulating the virtual objects.

[0021] Generating a 3D user interface and detecting user interactions with virtual objects may also include: detecting real-world objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects); tracking such real-world objects as they leave, enter, and move around the field of view in a video frame; and modifying or transforming such real-world objects as they are tracked. In various examples, different methods for detecting real-world objects and implementing such transformations may be used. For example, some examples may involve generating a 3D mesh model of one or more real-world objects, and implementing the transformation using transformations and animated textures of the model within a video frame. In other examples, tracking of points on real-world objects may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In another example, neural network analysis of video frames may be used to place images, models, or textures in content (e.g., images or video frames). Therefore, XR effect data may include both: images, models, and textures used to create transformations in content; and additional modeling and analysis information for implementing such transformations using real-world object detection, tracking, and placement.

[0022] Virtual objects can be UI elements such as buttons, sliders, input fields, etc., as well as other arbitrary objects of other kinds, such as puzzle pieces in a game or a cup of coffee. In many use cases, the user will be able to interact with these virtual objects using one or both of their hands based on hand tracking performed by the XR device. In some examples, the user can press a button, pinch and drag a slider to change a value, or pick up a cup of coffee and move it around in space. In order to indicate that the user can interact with a particular virtual object, that the virtual object provides an interactive action and the user's hand is close enough to interact with it, visual feedback and / or auditory feedback can be given to the user.

[0023] For example, in a puzzle game where a user can move a virtual puzzle piece with his or her hands, the puzzle piece may change its color, size, texture / appearance or may give audio feedback (play a specific sound) if the user's hand or finger is close enough to the puzzle piece in relative 3D space, thereby indicating that the intended user interface action can be performed. The user may then grasp the puzzle piece, for example, by using an appropriate gesture, such as moving the index finger toward the thumb in a grasping motion, which is thereby recognized by the XR device.

[0024] Further visual / audio feedback may then be given to indicate that the user is holding the virtual object and is interacting with it, such as dragging it around in the case of a virtual puzzle piece. In response to determining that the action is no longer being performed, such as by recognizing an appropriate gesture, such as the user moving their index finger away from their thumb in a release motion, the visual / audio feedback will cease. Providing feedback in this way improves accessibility and user experience, and improves the functionality of the XR system. This also mitigates the fact that the user does not receive tactile feedback as they would in the real world.

[0025] In some examples, a computer-implemented method for providing feedback in a display device including a camera is provided. The method includes: displaying a virtual object on the display device, tracking a user's hand in the field of view of the camera, determining the proximity of the user's hand to the virtual object, and providing user feedback based on the proximity of the user's hand to the virtual object. The method may also include: detecting a gesture performed by the user's hand when the user's hand is close to the virtual object, determining that the gesture can be used for the virtual object, and updating the state of the virtual object based on determining that the gesture can be used for the virtual object. Determining the proximity of the user's hand to the virtual object may include determining the intersection of the user's hand with a bounding box surrounding the virtual object.

[0026] The computer-implemented method may also include comparing the depth from the display device to the user's hand with the depth from the display device to the virtual object, and determining that the user's hand is not near the virtual object based on the depth from the display device to the user's hand not being within the range of the depth from the display device to the virtual object. A plurality of virtual objects may be displayed on the display device, each virtual object having an associated depth from the display device, the method further comprising comparing the depth from the display device to the user's hand with each of the depths of the plurality of virtual objects, and determining that the user's hand is not near a particular virtual object based on the depth from the display device to the user's hand not being within the range of the depth from the display device to the particular virtual object.

[0027] In some examples, the virtual object is an elongated virtual object, and the method further includes changing a graphical attribute of the elongated virtual object at a first position along the elongated virtual object corresponding to a proximity of a hand of a user, the hand of the user being a first hand of the user, tracking a second hand of the user in a field of view of a camera device, determining a proximity of the second hand of the user to the virtual object, and based on the proximity of the second hand of the user to the virtual object, changing the graphical attribute of the elongated virtual object at a second position along the elongated virtual object corresponding to a proximity of the second hand of the user.

[0028] Feedback may include audio feedback that increases in frequency (e.g., increases in pitch or repetition rate of a sound) as the user's hand approaches the virtual object. Feedback may also include changing visual attributes of other virtual objects near the virtual object.

[0029] In some examples, the virtual object includes a stack of subsidiary virtual objects, and the method also includes determining that a user's hand is near a specific subsidiary virtual object, and based on determining that the user's hand is near the specific virtual object, changing a property of the specific virtual object so that it is visible in the stack of subsidiary virtual objects or near the stack of subsidiary virtual objects.

[0030] A non-transitory computer-readable storage medium is also provided, which includes instructions that, when executed by a computer, cause the computer to perform operations for providing feedback in a display device including a camera device in accordance with any of the methods, method steps, and limitations described above, including but not limited to operations including: displaying a virtual object on the display device, tracking a user's hand in the field of view of the camera device, determining the proximity of the user's hand to the virtual object, and providing user feedback based on the proximity of the user's hand to the virtual object.

[0031] A computing device is also provided, comprising a processor and a memory storing instructions, which, when executed by the processor, configure the device to perform operations for providing feedback in a display device including a camera device according to any of the methods, method steps, and limitations described above, including but not limited to operations including: displaying a virtual object on the display device, tracking a user's hand in the field of view of the camera device, determining the proximity of the user's hand to the virtual object, and providing user feedback based on the proximity of the user's hand to the virtual object.

[0032] Other technical features will be readily apparent to those skilled in the art from the accompanying drawings, descriptions and claims.

[0033] Figure 1 is a perspective view of a head-mounted XR device (e.g., glasses 100) according to some examples. The glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or right optical element holder 106 connected by a bridge 112. A first optical element or left optical element 108 and a second optical element or right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0034] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or integrated construction.

[0035] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 302 discussed below.

[0036] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0037] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0038] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from a real-world scene.

[0039] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input may be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from a user of the glasses 100.

[0040] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0041] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0042] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 superimposed with the view of the real world seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 superimposed with the view of the real world seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of the glasses 100.

[0043] However, it should be understood that other display technologies or configurations may be used within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0044] In use, the user of the glasses 100 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 3 ) and / or hand movements, position, and orientation detected by the glasses 100 to interact with the glasses 100.

[0045] Figure 3is a block diagram showing a networked system 300 including details of the glasses 100 according to some examples. The networked system 300 includes the glasses 100, a user device 328, and a server system 332. The user device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using both a low power wireless connection 336 and / or a high speed wireless connection 334. The user device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The server system 332 may be a server system 332 that is a networked computing system that may be used separately. Fig.10 and Fig.11 The details of the software architecture 1004 or machine 1100 described in the implementation of the user device 328 and any elements of the server system 332 and network 330.

[0046] The glasses 100 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig.10 and Fig.11 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O components 1106, including output components 1128, motion components 1136, etc. Figure 2 Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the left and right eyes of a user.

[0047] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0048] The interface 312 refers to any source of user commands provided to the data processor 302. In one or more examples, the interface 312 is a physical button that, when pressed, sends a user input signal from the interface 312 to the low-power processor 314. The low-power processor 314 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 314 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from the camera 308. In other examples, the interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from the user device 328.

[0049] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to user device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0050] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from the user device 328 via the low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0051] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 302. High-speed processor 322 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.10 The high-speed processor 322, which executes the software architecture of the data processor 302, manages data transmission with the high-speed wireless circuit system 326, in addition to any other duties. In some examples, the high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0052] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, independent element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will direct high-speed processor 322 whenever a read or write operation involving memory 324 is desired.

[0053] The tracking module 340 estimates the position and orientation ("pose") of the glasses 100. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the positioning component 1140, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100 that indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0054] The GPU and display driver 338 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real environment.

[0055] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the user device 328 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1006, such as the messaging application 1046.

[0056] Figure 4 A user interface 400 of glasses 100 is depicted according to some examples. Glasses 100 provide a field of view 402 through which a real world 406, including a user's hand 408, can be perceived by the user. Glasses 100 capture a video feed generated by camera 308 that includes a plurality of image frames 410 of the real world 406. The captured image frames allow glasses 100 to locate and track the position and orientation (pose) of glasses 100 in the real world 406, as well as to identify and track objects in the real world, including the user's hand 408. The user's hand has an associated distance or depth A from glasses 100.

[0057] In some examples, glasses 100 generate a virtual hand mesh for one or both of the user's hands, the positioning and shape of which updates as the user moves and flexes their hands and fingers. The hand mesh can be used internally to determine hand proximity and gesture detection, and can also be used to provide a virtual overlay on some or all of the user's hands 408 to provide feedback to the user as discussed in more detail below.

[0058] Projected into the real world by the display 310 of the glasses 100 is a virtual object 404. The virtual object is positioned in a reference frame, which may be relative to the glasses 100, or fixed relative to the real world or one or more objects in the real world. The virtual object 404 has an associated depth or distance B from the glasses 100 in the reference frame. As much as possible, the distance of the virtual object 404 from the glasses 100 as perceived by the user of the glasses 100 is the same as the distance B from the glasses 100 in the associated virtual reference frame.

[0059] As discussed in more detail below, a user of the glasses can interact with a virtual object 404 by bringing the user's hand 408 into proximity with the virtual object 404. The exact nature of the available interactions, and the response of the glasses 100 or the networked system 300 to a particular interaction, is defined by the parameters and rules associated with the particular virtual object 404.

[0060] Figure 5A and Figure 5B Two examples of interactive virtual objects and their associated rules and properties are shown. The rules specify the conditions required to trigger feedback updates for the virtual objects. For example, Figure 5A , a virtual object in the nature of a 3D button 502 is defined, which can be pressed or clicked by moving a hand or finger or thumb thereon in a direction of increasing depth from the glasses 100. Associated with the 3D button 502 is a bounding box 504 that is larger than the 3D button 502 by a certain tolerance 506. Once it is determined that a portion of the user's hand (e.g., a finger or thumb) is positioned within the bounding box 504, appropriate feedback is provided, such as a change in the color or visibility of a hand mesh associated with a particular hand, or changing the color of the 3D button 502 to a specified color. The change in the visual attributes or characteristics of the 3D button 502 or other virtual object can be of any kind, including changing the overall color of a portion of a virtual object or hand mesh, providing a pattern or other decoration, changing the transparency of a virtual object or its visibility relative to other virtual objects, providing a halo effect, etc.

[0061] These visual cues indicate to the user that they are about to press or can press the 3D button 502, which state can also be reflected in the properties of the 3D button 502 (e.g., the properties are updated to reflect that it is in the "about to press" state). The color change or other visual feedback can be gradual, so that the closer the finger is to the 3D button 502, the more the color of the hand mesh or 3D button 502 changes toward the target color. This transition from the normal color to the target color occurs within a tolerance distance between the edge of the bounding box and the edge of the 3D button 502.

[0062] Once the finger is within the 3D button 502, another color is used on the 3D button 502 or the hand grid to indicate that the 3D button 502 is being pressed or has been pressed, and the properties of the 3D button 502 may be updated to a "pressed" state or a "pressed" state.

[0063] In addition to or in lieu of a color change, an acoustic feedback rule may be used. In some examples, once a finger or thumb or hand is positioned within the bounding box 504, a beep is turned on. The closer the finger is to the 3D button 502, the more constant or faster the beep. Once the finger is within the boundaries of the 3D button 502, a constant beep may be emitted (e.g., to indicate a "pressed state") or the sound may stop. In the latter case, another indication (color or another sound) may be used, or otherwise it may be apparent from the glasses 100 or networking system 300 response to the pressing of the 3D button 502, such as by the start of an action in the application 602 associated with pressing the 3D button 502 that the 3D button 502 has been pressed.

[0064] Figure 5B 508. An interactive rod-shaped virtual object is shown in FIG. 508. The interactive rod 508 is elongated and enhanced along its length, and can be enhanced in one or more portions to provide proximity feedback. The user can grasp or hold the interactive rod 508 with one or two hands. As before, a bounding box 510 is defined around the interactive rod 508, and the bounding box 510 is larger than the interactive rod 508 by a certain tolerance.

[0065] When the user's hand approaches the interactive stick 508 and intersects the bounding box 510, appropriate feedback is provided, such as a change in the color or visibility of a hand grid associated with the particular hand, or a change in the color of a feedback area 512 that includes a portion of the interactive stick 508 near the location of the user's hand.

[0066] These visual cues indicate to the user that they are about to interact with the interactive bar 508, and which states of these may also be reflected in the properties of the interactive bar 508 (e.g., updating the properties with an indication of the position of the user's hand along the interactive bar 508 to reflect that it is in a "ready to interact" state). The color change may be gradual, such that the closer the hand is to the interactive bar 508, the more the color of the hand grid or feedback area 512 changes toward the target color. As before, this transition from the normal color to the target color occurs within a tolerance distance between the edge of the bounding box and the edge of the interactive bar 508. In addition to or in lieu of the color change, an acoustic feedback rule may also be used. In some examples, once the hand is positioned within the bounding box 510, a beep is turned on. The closer the hand is to the interactive bar 508, the more constant or faster the beep will be.

[0067] Once the user's hand is within the bounding box 510, the user needs to make a grasping gesture to hold the interactive bar 508. The grasping gesture can be, for example, moving the hand from an open palm position to a closed finger position. Once the grasping gesture is detected, another color is used on the feedback area 512 or the hand grid to indicate that the interactive bar 508 has been grasped, and the attributes of the interactive bar 508 can be updated to a "grasp" or "holding" state using the identification of the user's grasping hand along the position of the interactive bar 508. This can also be applied to grasping the interactive bar 508 with two hands, in which case feedback is given in the same manner for the second hand and its position on the interactive bar 508, and in the case of grasping with two hands, the attributes of the interactive bar 508 will be updated to reflect that the interactive bar 508 is held by two hands and the position along which the interactive bar 508 is held.

[0068] Figure 6 A UI architecture 600 is shown that can be used to implement feedback according to some examples. The UI architecture 600 can be embodied in the user device 328, the glasses 100, embodied on the server system 332, or embodied in some combination thereof. The UI architecture 600 includes an application 602, a feedback manager 604, a hand tracking engine 606, and a gesture detection engine 608.

[0069] The application 602, after being launched at block 614, performs functions associated with its intended augmented reality functionality (e.g., gaming, content capture, where an AR overlay or object is displayed to a user on or embedded in the real world). In particular, in block 616, the application 602 creates and displays an augmented reality object in 3D space for display to a user of the glasses 100. The application 602 also includes virtual object properties and rules, including the current shape, positioning, and orientation of the virtual object, the types of interactions that can be performed on or with the object, the types of gestures available, such as grasping or button pressing, bounding box shape and tolerance, etc. In block 618, these properties and rules are provided to the feedback manager 604, where the object and its properties and rules are registered.

[0070] The application 602 also receives information about changes in the state of the object based on the user interaction, and in block 620, the information is reported to the application 602 by the feedback manager 604. The change in state can be, for example, a hand or finger within the bounding box 504, a hand or finger moving to or from a virtual object in a particular direction, a position on a virtual object or in the bounding box of the hand or finger, and any related gestures that have been detected. In block 620, the feedback manager 604 also applies feedback rules for a particular virtual object and notifies the application 602 of visual feedback or auditory feedback to be applied to the virtual object.

[0071] In response to receiving the change in state and associated feedback from the feedback manager 604, the application 602 visually indicates the change in state of the virtual object to the user of the glasses 100 in the display of the virtual object on the display 310, or acoustically indicates the change in state of the virtual object to the user of the glasses 100 via the associated audio transducer. The change in state of the virtual object and the associated movement of the associated hand or finger are also reported to the application 602 so that the application 602 can take action in the application 602 itself corresponding to the change in state and the change in hand positioning, for example, grasping and swinging a virtual tennis racket.

[0072] Feedback manager 604 includes interactivity manager 610 and accessibility assistant 612. Interactivity manager 610 includes proximity functionality including algorithms that determine the proximity of the user's hand 408 and fingers to the virtual object 404 and the closest point on / to the surface of the virtual object 404 to the user's hand 408 or fingers (one or more fingers, one or more thumbs). This is done based on data provided from hand tracking engine 606, as is conventional and well known in the art.

[0073] The interactivity manager 610 also includes an interactivity function, which is an algorithm that checks whether a particular virtual object 404 provides interactivity and the range and angle relative to the object within which one or more different interactive actions can be performed. The interactivity manager 610 also checks whether any relevant interactive actions are being performed on the virtual object 404 (or more than one virtual object at the same time using one or two hands). This is done in conjunction with the gesture detection engine 608, which detects and reports whether the user is performing a known gesture. The gesture detection engine 608 is also conventional and well known in the art. The detected gesture reported by the gesture detection engine 602 is checked by the interactivity manager 610 against a list of gestures available for a particular virtual object 404.

[0074] In use, the interactivity manager 610 receives hand tracking data from the hand tracking engine 606 and gesture data from the gesture detection engine 608. The interactivity manager 610 first determines whether the user's hand (or finger) is within the bounding box of the virtual object. If so, this fact is reported to the accessibility assistant 612 along with the location of the hand or finger and the direction of movement of the hand or finger (if any). The interactivity manager 610 secondly determines whether the relevant gesture reported by the gesture detection engine 608 for the virtual object 404 is being performed when the hand or finger is within the bounding box. If so, this fact is also reported to the accessibility assistant 612.

[0075] The interactivity manager 610 also monitors the relevant ongoing actions or gestures performed by the user's hand while in the vicinity of the virtual object based on the updated data from the hand tracking engine 606 and the gesture detection engine 608. For example, if the virtual object is a virtual object that can be grasped, moved, and released, then when the gesture is performed within the bounding box of the virtual object, the interactivity manager 610 will report a "grasp" state to the accessibility assistant 612, if the relevant hand moves, then the interactivity manager 610 will report hand positioning updates for the relevant hand, and if a "release" gesture is detected, then the "release" state and release positioning and orientation will be reported to the accessibility assistant 612.

[0076] The accessibility assistant 612 receives hand or finger positioning information and movement direction, virtual object state, and any detected related gestures from the interactivity manager 610. The accessibility assistant 612 determines appropriate feedback based on the virtual object state, any related gestures, and any associated movement of the user's hand. The accessibility assistant 612 then informs the application 602 of the feedback to be provided based on the state and information reported from the interactivity manager 610 by applying rules and properties related to the specific virtual object.

[0077] Fig. 7A and Figure 7B User feedback provided in an AR wearable device, such as glasses 100, according to some examples is shown. As shown in the figure, a deck 702 of stacked UI virtual objects 704 is displayed to a user of the glasses 100. The deck 702 is perceived by the user as being located in a 3D space in a real-world environment in front of the user. The deck 702 of UI virtual objects 704 is arranged along a depth direction 708 and is perceived by the user as being along the depth direction 708, which extends at least partially toward or away from the user and the glasses 100. Also shown in the figure is a user's hand 706, which is the user's actual hand as seen through the glasses 100, although it itself may be enhanced, for example with a grid or other overlay. In Fig. 7A , the user's hand 706 is far enough away from the layer group 702 so that any related gestures performed by the hand do not affect any UI virtual object in the layer group 704. That is, the user's hand 706 is outside any bounding box associated with the layer group 702 or any UI virtual object in the UI virtual object 704. The layer group 702 or any content in its content is not highlighted.

[0078] In this case, the layer group 702 can be regarded as a virtual object including multiple child virtual objects, namely the UI virtual object 704. In such a case, the layer group 702, as a whole and each UI virtual object in the UI virtual object 704, can have feedback associated with it. For example, the layer group can be provided with auditory feedback, while each UI virtual object 704 is provided with visual feedback. Alternatively or in addition, the visual feedback for the layer group 702 can be based on a bounding box that extends further than the edge of the layer group 702 compared to the edge of the UI virtual object 704. In such a case, feedback for the layer group 702 will be provided when the user's hand is slightly far away, and feedback for a specific UI virtual object 704 will begin when the user's hand is closer.

[0079] exist Figure 7B In one example, the user's hand has approached the layer group 702, or more accurately, the user's hand has approached a position in the real world that corresponds to the position of the layer group 702 in a 3D model of the surrounding environment, the 3D model including at least the position of one or more objects in the real world. The user's hand has intersected with the bounding box around the layer group 702 or around one of the UI virtual objects 704, and a specific UI virtual object 710 corresponding to the depth of the hand 706 along the layer group 702 has been highlighted to indicate that it is a UI virtual object that can be used for selection or interaction. The visibility of the UI virtual object 710 has also changed relative to other UI virtual objects in the UI virtual objects 704, so that the camera icon on the UI virtual object 710 is visible to the user.

[0080] If the user now moves his hand 706 along the edge of the layer group 702 toward 712 or away from 714 the glasses 100 along the depth direction 708, the UI virtual object 704 corresponding to the new depth of the user's hand will be highlighted and made visible, and Fig. 7A As shown, the UI virtual object 710 is "returned" to the layer group 702 by restoring its display properties. The user can thus scroll along the layer group to identify the UI virtual object 704 with which they wish to interact.

[0081] Figure 7BThe UI virtual object 710 highlighted in the figure can be obtained or activated by an appropriate gesture performed by the user's hand. For example, a tapping action using the user's index finger can activate the UI virtual object 710 or a function associated with it, while a gripping action in which the thumb and index finger are together can grip the object and allow it to be dragged to a different position, where the object can be released by separating the thumb and index finger. The specific response to the gesture will of course depend on the implementation and function of the application 602. In addition, as mentioned above, different tones or beeps can be provided in place of or in addition to visual effects.

[0082] In some examples, the layer group 702 has a bounding box associated with it, rather than a bounding box for a single UI virtual object 704, and instead, the UI virtual object 704 has a depth associated with it within the bounding box. The depth within the bounding box of the layer group 702 is used to identify the UI virtual object 704 at the depth corresponding to the positioning of the user's hand 706, to which feedback will be applied. This concept can be applied more widely by maintaining an index of the depth of the virtual objects 404, which can be compared with the depth of the user's hand 408 for each virtual object in the virtual objects 404. When determining whether there are any virtual objects 404 that the user can interact with, all virtual objects in the index that are not at the depth of one of the user's hands 408 can be rejected for further consideration. Then, the virtual object 404 at the depth of one of the user's hands 408 can be checked to see if one of the user's hands 408 is in the bounding box for feedback and whether any relevant gestures are being performed.

[0083] In other examples, adjacent or surrounding virtual objects 404 near the interactive virtual object are also enhanced or highlighted to provide additional feedback. The feedback applied to the nearby virtual objects may not be as obvious as the feedback provided on or to the optional virtual objects. For example, in a grid or array of virtual objects, in addition to highlighting the optional virtual objects, the virtual objects surrounding the optional virtual objects may also be highlighted, but to a lesser extent, or a lighter highlight color shade than the color applied to the virtual object 404 may be applied. This provides additional feedback to help the user select the correct virtual object.

[0084] Figure 8 800 is a flowchart illustrating a method of providing feedback according to some examples. For the purpose of illustration, the operations of flowchart 800 are described herein as occurring serially or linearly. However, multiple operations of flowchart 800 may occur in parallel. In addition, the operations of flowchart 800 do not need to be performed in the order shown and / or one or more blocks of flowchart 800 do not need to be performed and / or may be replaced by other operations.

[0085] Figure 8 The operations shown in the flowchart 800 will generally be performed on the glasses 100. In other examples, the operations are performed jointly between an application running on the user device 328 and a data processor 302 and related hardware in or associated with the glasses 100. For the purpose of clarity, the flowchart 800 is discussed herein with reference to such an example. Various implementations are of course possible in which some of the operations occur in the server system 332, or in which one application calls another application or SDK to obtain the desired functionality.

[0086] The method begins at operation 802, where the glasses 100, in conjunction with the application 602, display one or more virtual objects 404 on the display 310 of the glasses 100. The nature of the display of the virtual objects 404 is determined by the functionality of the application 602, user input, the positioning and orientation of the glasses 100 in the real world, the positioning and orientation of objects or people in the real world, etc. The application 602 also registers feedback rules for each virtual object 404 with the feedback manager 604, and provides updated positioning information for the virtual objects 404 based on the functionality of the application 602 as the glasses 100 move in the environment. In this case, the virtual objects 404 are virtual objects 404 that the user can interact with using gestures, rather than virtual objects that are invalid for gestures.

[0087] In operation 804 , the location of at least one of the user's hands 408 is detected by the hand tracking engine 606 and reported to the feedback manager 602 , and in operation 806 , the feedback manager 602 compares the location of the user's hand 408 with the location of the virtual object 404 .

[0088] In some examples, this can be a two-step process where at depth A of the user's hand 408 (see Figure 4 ) is compared to the depth B of each virtual object in the virtual objects 404. If the depth A of the user's hand 408 is not within the depth corresponding to the bounding box of one of the virtual objects 404 (e.g., the bounding box 504 or the bounding box 510), the user's hand 408 cannot intersect the bounding box. In such a case, the test in operation 808 fails, and the method returns to operation 802, where the display of the virtual objects 404 is continued. If any feedback has been previously provided based on the intersection of the user's hand 408 with the virtual objects 404, the feedback stops in operation 820 because the hand no longer intersects any bounding box.

[0089] If the depth A of the user's hand 408 is within the range of the depth corresponding to the bounding box of one of the virtual objects 404 (e.g., bounding box 504 or bounding box 510), the user's hand 408 may intersect the bounding box. In such a case, any virtual objects that are not at the relevant depth are eliminated from further consideration. The positioning of the user's hand 408 relative to any remaining virtual objects 404 is checked in a plane perpendicular to the depth direction (i.e., in the x and y directions if z is the depth direction) at the particular depth A of the user's hand 408. If the positioning of the user's hand is not within the bounding box of the remaining virtual objects 404 based on this comparison, the test in operation 808 fails and the method returns to operation 802, where the display of the virtual objects 404 continues.

[0090] If the positioning of the user's hand is within the bounding box of the remaining virtual objects 404, the test in operation 808 passes, and the method continues at operation 810. In operation 810, as discussed in more detail above, visual or auditory feedback is provided to the user by the glasses 100 to indicate that the user's hand 408 is in a position to interact with a particular virtual object 404. This can be achieved by the interactivity manager 610 reporting to the accessibility assistant 612 that the user's hand 408 intersects the bounding box of the particular virtual object 404. The accessibility assistant 612 determines the appropriate feedback by applying the rules and properties associated with the particular virtual object, and then the accessibility assistant 612 notifies the application 602 of the feedback to be provided. The feedback is then provided by the glasses 100 as defined and directed by the application 602.

[0091] Concurrently with these operations, the gesture detection engine 608 reports any detected gestures to the feedback manager 604. If no gestures are detected in operation 812, the method returns to operation 808 where it is checked whether the hand still intersects the bounding box of the associated virtual object 404, and whether the hand intersects the bounding box of any other virtual object, and the method continues from operation 808.

[0092] If the gesture is detected and reported to the interactivity manager 610 in operation 812, then in operation 814 the interactivity manager 610 checks to see if the gesture is available for the relevant virtual object 404 whose bounding box is intersecting the hand 408 of the user performing the gesture. If the gesture is not available for the particular virtual object 404, then the method returns to operation 808 and continues from operation 808.

[0093] If the detected gesture is applicable to the relevant virtual object as determined in operation 814, the interactivity manager 610 updates the state of the relevant virtual object 404 (e.g., to a "grasped" state or a "button pressed" state) in operation 816, where applicable. The interactivity manager 610 then reports the hand (or finger) positioning information and movement direction received from the hand tracking engine 606, the updated virtual object state, and the relevant gesture to the accessibility assistant 612. The accessibility assistant 612 determines any appropriate feedback based on the virtual object state, the relevant gesture, and any associated movement of the user's hand by applying rules and properties related to the specific virtual object. The accessibility assistant 612 then notifies the application 602 of the feedback to be provided based on the state and information (if any) reported from the interactivity manager 610. The virtual object state, the relevant gesture, and any associated movement of the user's hand are also reported to the application 602, which then takes relevant actions in operation 818 based on the functionality of the application 602.

[0094] Fig. 9 is a block diagram illustrating an example messaging system 900 for exchanging data (e.g., messages and associated content) over a network. The messaging system 900 includes multiple instances of user devices 328 that host several applications including a messaging client 902 and other applications 904. The messaging client 902 is communicatively coupled to other instances of the messaging client 902 (e.g., hosted on respective other user devices 328), a messaging server system 906, and a third-party server 908 via a network 330 (e.g., the Internet). The messaging client 902 can also communicate with the locally hosted application 904 using an application program interface (API).

[0095] The messaging clients 902 are able to communicate and exchange data with other messaging clients 902 and with a messaging server system 906 via the network 330. The data exchanged between the messaging clients 902 and between the messaging clients 902 and the messaging server system 906 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0096] The messaging server system 906 provides server-side functionality to a particular messaging client 902 via the network 330. Although some functions of the messaging system 900 are described herein as being performed by the messaging client 902 or by the messaging server system 906, it may be a design choice whether some functions are located within the messaging client 902 or within the messaging server system 906. For example, it may be technically preferred to initially deploy some technologies and functions within the messaging server system 906, but then migrate the technologies and functions to the messaging client 902 where the user device 328 has sufficient processing power.

[0097] The messaging server system 906 supports various services and operations provided to the messaging client 902. Such operations include transmitting data to the messaging client 902, receiving data from the messaging client 902, and processing data generated by the messaging client 902. As examples, the data may include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 900 is activated and controlled by functions available via the user interface (UI) of the messaging client 902.

[0098] Turning now specifically to the messaging server system 906, an application program interface (API) server 910 is coupled to and provides a programming interface to an application server 914. The application server 914 is communicatively coupled to a database server 916, which facilitates access to a database 920 that stores data associated with messages processed by the application server 914. Similarly, a web server 924 is coupled to and provides a web-based interface to the application server 914. To this end, the web server 924 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0099] The application program interface (API) server 910 receives and transmits message data (e.g., commands and message payloads) between the user device 328 and the application server 914. Specifically, the application program interface (API) server 910 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 902 to activate the functions of the application server 914. The application program interface (API) server 910 exposes various functions supported by the application server 914, including: account registration; login functionality; sending messages from a particular messaging client 902 to another messaging client 902 via the application server 914, sending media files (e.g., images or videos) from the messaging client 902 to the messaging server 912, and possible access by another messaging client 902; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the user device 328; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 902).

[0100] The application server 914 hosts several server applications and subsystems, including, for example, a messaging server 912, an image processing server 918, and a social network server 922. The messaging server 912 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 902. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 902. In view of the hardware requirements for processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 912.

[0101] The application server 914 also includes an image processing server 918 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 912.

[0102] The social network server 922 supports various social networking functions and services and makes these functions and services available to the messaging server 912. To this end, the social network server 922 maintains and accesses an entity graph within the database 920. Examples of functions and services supported by the social network server 922 include identifying other users in the messaging system 900 with whom a particular user has a relationship or who the particular user is "following", and also include identifying interests and other entities of a particular user.

[0103] The messaging client 902 may notify the user of the user device 328 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 902 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 902 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0104] Fig.10 1000 is a block diagram illustrating a software architecture 1004 that may be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware, such as a machine 1002 including a processor 1020, a memory 1026, and an I / O component 1038. In this example, the software architecture 1004 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1004 includes layers such as an operating system 1012, a library 1008, a framework 1010, and an application 1006. In operation, the application 1006 invokes an API call 1050 through the software stack and receives a message 1052 in response to the API call 1050.

[0105] The operating system 1012 manages hardware resources and provides public services. The operating system 1012 includes, for example, a core 1014, services 1016, and drivers 1022. The core 1014 serves as an abstraction layer between the hardware and other software layers. For example, the core 1014 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1016 can provide other public services to other software layers. Drivers 1022 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1022 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0106] The libraries 1008 provide low-level common infrastructure used by the applications 1006. The libraries 1008 may include system libraries 1018 (eg, C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1008 may include an API library 1024, such as a media library (e.g., a library for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting graphics content in two dimensions (2D) and three dimensions (3D) on a display, GLMotif for implementing a 3D user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1008 may also include various other libraries 1028 to provide many other APIs to the application 1006.

[0107] The framework 1010 provides a high-level common infrastructure used by the applications 1006. For example, the framework 1010 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1010 can provide a wide range of other APIs that can be used by the applications 1006, some of which can be specific to a particular operating system or platform.

[0108] In an example, applications 1006 may include a home application 1036, a contacts application 1030, a browser application 1032, a book reader application 1034, a location application 1042, a media application 1044, a messaging application 1046, a game application 1048, and a variety of other applications such as third-party applications 1040. Applications 1006 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1006 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1040 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROIDTM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1040 can call the API call 1050 provided by the operating system 1012 to facilitate the functions described in this article.

[0109] Fig.11 The invention is a diagrammatic representation of a machine 1100 in which instructions 1110 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 1100 to perform any one or more of the methods discussed herein. For example, the instructions 1110 may cause the machine 1100 to perform any one or more of the methods described herein. The instructions 1110 transform a general-purpose, unprogrammed machine 1100 into a specific machine 1100 that is programmed to perform the functions described and illustrated in the manner described. The machine 1100 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1110 that specify actions to be taken by the machine 1100, either sequentially or otherwise. In addition, while a single machine 1100 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 1110 to perform any one or more of the methodologies discussed herein.

[0110] The machine 1100 may include a processor 1102, a memory 1104, and an I / O component 1106, which may be configured to communicate with each other via a bus 1144. In an example, the processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1108 that executes instructions 1110 and a processor 1112. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.11 Multiple processors 1102 are shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0111] The memory 1104 includes a main memory 1114, a static memory 1116, and a storage unit 1118, all of which are accessible by the processor 1102 via the bus 1144. The main memory 1104, the static memory 1116, and the storage unit 1118 store instructions 1110 that implement any one or more of the methods or functions described herein. During execution of the instructions 1110 by the networked system 300, the instructions 1110 may also reside, in whole or in part, within the main memory 1114, within the static memory 1116, within the machine-readable medium 1120, within the storage unit 1118, within one or more of the processors 1102 (e.g., within a cache memory of a processor), or within any suitable combination thereof.

[0112] I / O components 1106 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1106 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 1106 may include Fig.111106. In various examples, the I / O components 1106 may include output components 1128 and input components 1132. The output components 1128 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1132 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0113] In another example, the I / O component 1106 may include: a biometric component 1134, a motion component 1136, an environmental component 1138, or a positioning component 1140, as well as various other components. For example, the biometric component 1134 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1136 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1138 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1140 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0114] A variety of technologies may be used to achieve communication. I / O components 1106 also include communication components 1142 that are operable to couple networked system 300 to network 1122 or device 1124 via coupling 1130 and coupling 1126, respectively. For example, communication components 1142 may include a network interface component or another suitable device that interfaces with network 1122. In other examples, communication components 1142 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Device 1124 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0115] In addition, the communication component 1142 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1142 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1142, such as a location obtained via Internet Protocol (IP) geolocation, ... Location obtained by signal triangulation, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0116] Various memories (e.g., memory 1104, main memory 1114, static memory 1116, and / or memory of processor 1102) and / or storage unit 1118 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1110) when executed by processor 1102 cause various operations to implement the disclosed examples.

[0117] The instructions 1110 may be transmitted or received over the network 1122 via a network interface device (e.g., a network interface component included in the communications component 1142) using a transmission medium and using any of a number of well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1110 may be transmitted or received to the device 1124 via a coupling 1126 (e.g., a peer-to-peer coupling) using a transmission medium.

[0118] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0119] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0120] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0121] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0122] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0123] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media".

[0124] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0125] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0126] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method for providing feedback in a display device including a camera, the method comprising: displaying a virtual object on the display device; Tracking a user's hand in the field of view of the camera; determining a proximity of the user's hand to the virtual object; as well as User feedback is provided based on the proximity of the user's hand to the virtual object.

2. The computer-implemented method of claim 1 , further comprising: detecting a gesture performed by a hand of the user while the hand of the user is proximate to the virtual object; determining that the gesture is applicable to the virtual object; as well as Based on determining that the gesture is applicable to the virtual object, a state of the virtual object is updated.

3. The computer-implemented method of claim 1 , wherein: Determining the proximity between the user's hand and the virtual object includes: An intersection of the user's hand with a bounding box surrounding the virtual object is determined.

4. The computer-implemented method of claim 1 , further comprising: comparing a depth from the display device to the user's hand with a depth from the display device to the virtual object; as well as Based on the fact that the depth from the display device to the user's hand is not within the range of the depth from the display device to the virtual object, it is determined that the user's hand is not near the virtual object.

5. The computer-implemented method of claim 1 , further comprising: displaying a plurality of virtual objects on the display device, each virtual object having an associated depth from the display device; comparing a depth from the display device to the user's hand to each of the depths of the plurality of virtual objects; as well as Based on the fact that the depth from the display device to the user's hand is not within the range of the depth from the display device to the specific virtual object, it is determined that the user's hand is not near the specific virtual object.

6. The computer-implemented method of claim 1 , wherein: The virtual object is an elongated virtual object, and the computer-implemented method further comprises: changing a graphical attribute of the elongated virtual object at a first position along the elongated virtual object corresponding to a proximity of a hand of the user, the hand of the user being a first hand of the user; Tracking a second hand of the user in the field of view of the camera; determining a proximity of the user's second hand to the virtual object; and Based on the proximity of the user's second hand to the virtual object, a graphical attribute of the elongated virtual object is changed at a second position along the elongated virtual object corresponding to the proximity of the user's second hand.

7. The computer-implemented method of claim 6, wherein: The virtual object includes a stack of child virtual objects, and the method further includes: Determining that the user's hand is near a specific sub-virtual object; and Based on determining that the user's hand is near the specific virtual object, a property of the specific virtual object is changed to make the specific virtual object visible in or near the stack of child virtual objects.

8. The computer-implemented method of claim 1 , wherein: The feedback includes audio feedback that increases in frequency as the user's hand approaches the virtual object.

9. The computer-implemented method of claim 1 , wherein: The feedback includes changing visual attributes of other virtual objects in the vicinity of the virtual object.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations for providing feedback in a display device including a camera, the operations comprising: displaying a virtual object on the display device; Tracking a user's hand in the field of view of the camera; determining a proximity of the user's hand to the virtual object; as well as User feedback is provided based on the proximity of the user's hand to the virtual object.

11. The non-transitory computer-readable storage medium of claim 10, wherein: Determining the proximity between the user's hand and the virtual object includes: An intersection of the user's hand with a bounding box surrounding the virtual object is determined.

12. The non-transitory computer-readable storage medium of claim 10, wherein: The operations also include: comparing the depth from the display device to the user's hand with the depth from the display device to the virtual object; and Based on the fact that the depth from the display device to the user's hand is not within the range of the depth from the display device to the virtual object, it is determined that the user's hand is not near the virtual object.

13. The non-transitory computer-readable storage medium of claim 10, wherein: The operations also include: displaying a plurality of virtual objects on the display device, each virtual object having an associated depth from the display device; comparing a depth from the display device to the user's hand with each of the depths of the plurality of virtual objects; and Based on the fact that the depth from the display device to the user's hand is not within the range of the depth from the display device to the specific virtual object, it is determined that the user's hand is not near the specific virtual object.

14. The non-transitory computer-readable storage medium of claim 10, wherein: The virtual object is a long virtual object, and the operation further includes: changing a graphical attribute of the elongated virtual object at a first position along the elongated virtual object corresponding to a proximity of a hand of the user, the hand of the user being a first hand of the user; Tracking a second hand of the user in the field of view of the camera; determining a proximity of the user's second hand to the virtual object; and Based on the proximity of the user's second hand to the virtual object, a graphical attribute of the elongated virtual object is changed at a second position along the elongated virtual object corresponding to the proximity of the user's second hand.

15. The non-transitory computer-readable storage medium of claim 10, wherein: The feedback includes changing visual attributes of other virtual objects in the vicinity of the virtual object.

16. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for providing feedback in a display device including a camera, the operations comprising: displaying a virtual object on the display device; Tracking a user's hand in the field of view of the camera; determining a proximity of the user's hand to the virtual object; and User feedback is provided based on the proximity of the user's hand to the virtual object.

17. The computing device of claim 16, wherein: The operations also include: detecting a gesture performed by a hand of the user when the hand of the user is near the virtual object; determining that the gesture is applicable to the virtual object; and Based on determining that the gesture is applicable to the virtual object, a state of the virtual object is updated.

18. The computing device of claim 16, wherein: The operations also include: displaying a plurality of virtual objects on the display device, each virtual object having an associated depth from the display device; comparing a depth from the display device to the user's hand with each of the depths of the plurality of virtual objects; and Based on the fact that the depth from the display device to the user's hand is not within the range of the depth from the display device to the specific virtual object, it is determined that the user's hand is not near the specific virtual object.

19. The computing device of claim 16, wherein: Determining the proximity between the user's hand and the virtual object includes: An intersection of the user's hand with a bounding box surrounding the virtual object is determined.

20. The computing device of claim 16, wherein: The feedback includes audio feedback that increases in frequency as the user's hand approaches the virtual object.