Hand-tracked text selection and modification

Abstract

An augmented reality (AR) system is provided. The AR system generates a text scene object including a virtual text object based on the text, and detects a text selection gesture made by a user of the AR system. The AR system generates a selection line based on a landmark of a user's hand. The AR system detects a confirmation gesture made by the user and sets a text selection start point at an intersection of a selection line and a starting virtual text object of the one or more virtual text objects. The AR system detects a subsequent confirmation gesture made by the user, sets a text selection end point, and selects a selected text from the text based on the text selection start point and the text selection end point.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 17 / 664,010, filed on May 18, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to user interfaces and, more particularly, to user interfaces used in augmented reality and virtual reality. Background Art

[0004] A head-mounted device may be implemented with a transparent or translucent display through which a user of the head-mounted device can view the surrounding environment. Such a device enables a user to look through a transparent or translucent display to view the surrounding environment, and also to see objects (e.g., virtual objects such as presentations of 2D or 3D graphic models, images, videos, text, etc.) generated for display to appear as part of the surrounding environment and / or superimposed on the surrounding environment. This is commonly referred to as "augmented reality" or "AR". A head-mounted device may also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is commonly referred to as "virtual reality" or "VR". As used herein, unless the context indicates otherwise, the term AR refers to either or both of augmented reality and virtual reality as traditionally understood.

[0005] Users of head mounted devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with a user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or action, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 yes Figure 1 Additional views of a head mounted device according to some examples.

[0009] Figure 3 is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0010] Figure 4Ais a process flow diagram of text selection and modification processing according to some examples.

[0011] Figure 4B Gestures according to some examples are shown.

[0012] Figure 4C , Figure 4D and Figure 4E is an illustration of an AR-augmented scene including a text scene object according to some examples.

[0013] Figure 5A is a process flow diagram of text selection and modification processing according to some examples.

[0014] Figure 5B is another illustration of a gesture according to some examples.

[0015] Figure 5C , Figure 5D and Figure 5E is an illustration of another AR-augmented scene including a text scene object according to some examples.

[0016] Fig. 6A is a process flow diagram of virtual keyboard processing according to some examples.

[0017] Figure 6B is an illustration of a physical surface selection process according to some examples.

[0018] Figure 6C is an illustration of a virtual keyboard user interface according to some examples.

[0019] Figure 7 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some examples.

[0020] Figure 8 According to some examples Figure 1 A block diagram of the details of the head mounted device.

[0021] Fig. 9 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples. DETAILED DESCRIPTION

[0022] AR systems are limited when it comes to available user input modalities. It is more complicated for users of AR systems to indicate user intent and invoke actions or applications compared to other mobile devices such as mobile phones. When using a mobile phone, a user can go to the home screen and click on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easy to perform on AR systems. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it is desirable to have an input modality that allows a wider variety of inputs that can be used by users to indicate their intent through user input.

[0023] An input modality that can be used with an AR system is hand tracking combined with direct manipulation of virtual objects (DMVO), where a user interface is provided to the user, which is displayed to the user in an AR overlay with 2D or 3D rendering. The rendering has the properties of a graphical model in 2D or 3D, where virtual objects located in the model correspond to interactive elements of the user interface. In this way, the user perceives the virtual objects as objects within an overlay in the user's field of view of the real world scene when wearing the AR system, or perceives the virtual objects as objects within the virtual world viewed by the user when wearing the AR system. In order to enable the user to manipulate virtual objects, the AR system detects the user's hands and tracks their movement, position, and / or location to determine the user's interaction with the virtual objects.

[0024] Gestures that do not involve DMVO provide another input modality suitable for AR systems. Gestures are made by the user moving and positioning parts of the user's body, and when the user wears the AR system, those parts of the user's body can be detected by the AR system. The detectable parts of the user's body may include parts of the user's upper torso, arms, hands, and fingers. The composition of the gesture may include the movement of the user's arms and hands, the position of the user's arms and hands in the real-world scene environment, and the positioning of the user holding his upper torso, arms, hands, and fingers. Gestures are useful in providing AR experiences for users because they provide a way to provide user input to the AR system during the AR experience without causing the user to shift their attention away from the AR experience. As an example, in an AR experience that is an operating manual for a piece of machinery, the user can simultaneously view the piece of machinery in the real-world scene through the lens of the AR system, view the AR overlay on the real-world scene view of the machinery, and provide user input to the AR system.

[0025] Head tracking information provides another input modality suitable for use with an AR system. The AR system continuously collects and uses updated sensor data describing the movement of the AR system to determine changes in relative positioning and orientation with respect to physical objects in the real-world scene environment. The sensor data allows the AR device to detect the user's attention to virtual objects and physical objects within the user's field of view.

[0026] An improved text input user interface is provided to users of an AR system by combining the input modalities of hand tracking DMVO, gestures, and head tracking. The AR system uses a combination of gestures and DMVO methods to provide users with selection and modification of text within a text scene object of an AR experience. The user makes a gesture to open or invoke the text selection and modification user interface, and makes another gesture to start selecting text from a text scene object. The AR system provides the user with a virtual stylus. The user uses the virtual stylus to select the text to be modified. With a free hand, the user makes a gesture to set the text selection point at the location where the virtual stylus points.

[0027] In some examples, instead of a virtual stylus, the AR system detects the user's focus point to select a portion of the text to be modified. The user focuses on the portion of the text where the user wants to set an endpoint, and the user makes a gesture to set a text selection point that defines the selected text.

[0028] In some examples, the AR system projects a virtual keyboard onto a physical surface, and the user interacts with the virtual keyboard and with text displayed in a text scene object. The physical surface provides tactile feedback to the user.

[0029] Other technical features may be readily apparent to those skilled in the art from the accompanying drawings, descriptions and claims.

[0030] Figure 1 According to some examples, a head mounted device (e.g., Figure 1 100). The glasses 100 may include a frame 102 made of any suitable material such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0031] Frame 102 also includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0032] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be at least partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 802 discussed below.

[0033] The computer 120 also includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0034] The glasses 100 include a first camera or left camera 114 and a second camera or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may also include biometric sensors, position sensors, motion sensors, etc.

[0035] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real-world scene.

[0036] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 120 and the right temple piece 122. The touchpad 126 is generally arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than this. Additional user input can be provided by one or more buttons 128, which, in the example shown, are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a device by which the glasses 100 can receive input from the user of the glasses 100.

[0037] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0038] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0039] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 superimposed with a view of the real world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 superimposed with a view of the real world scene seen by the user. The combination of the GPU, front optical assembly 202, left optical element 108, and right optical element 110 provides an optical engine of glasses 100. Glasses 100 use the optical engine to generate an overlay with the user's view of the real world scene, including displaying a user interface to the user of glasses 100.

[0040] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to a user in the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.

[0041] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, voice input, or on an associated device (e.g., Figure 8 The user may interact with the glasses 100 based on touch input on a client device 826 shown in FIG. 1 and / or hand movement, position, and orientation detected by the glasses 100 .

[0042] Figure 3 is a diagrammatic representation of a computing device 300 in which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed that cause the computing device 300 to perform any one or more of the methodologies discussed herein. The computing device 300 may be used as Figure 1 The computer 120 of the glasses 100. For example, the instructions 310 can cause the computing device 300 to perform any one or more of the methods described herein. The instructions 310 transform the general non-programmed computing device 300 into a specific computing device 300 that performs the functions described and shown in the described manner. The computing device 300 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the computing device 300 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The computing device 300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web (network) device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 310 specifying actions to be taken by the computing device 300. In addition, while a single computing device 300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute the instructions 310 to perform any one or more of the methodologies discussed herein.

[0043] Computing device 300 may include processor 302, memory 304, and I / O components 306 that may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but computing device 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0044] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. The instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320 within the storage unit 318, within one or more of the processors 302 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the computing device 300.

[0045] The I / O components 306 may include various components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be appreciated that the I / O components 306 may include Figure 3306. In various examples, the I / O components 306 may include output components 328 and input components 332. The output components 328 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 332 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0046] In some examples, I / O components 306 may include biometric components 334, motion components 336, environment components 338, and positioning components 340, as well as various other components. For example, biometric components 334 include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. Motion components 336 may include an inertial measurement unit, an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental components 338 include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning components 340 may include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., an altimeter or barometer from which the air pressure is detected and the altitude can be obtained), orientation sensor components (e.g., an inertial measurement unit (IMU)), etc.

[0047] Various technologies may be used to implement communications. I / O component 306 also includes a communication component 342 that is operable to couple computing device 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication component 342 may include a network interface component or other suitable device that interfaces with network 322. In other examples, communication component 342 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low energy consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0048] In addition, the communication component 342 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data symbol (Dataglyph), a MaxiCode, PDF417, an UltraCode, a UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for an audio signal for identifying a tag). In addition, various information can be obtained via the communication component 342, such as a location obtained via an Internet Protocol (IP) geolocation, a location obtained via a NFC smart tag detection component, a location obtained via a NFC smart tag detection component, and a location obtained via a NFC smart tag detection component. Location derived from signal triangulation, location derived via detection of NFC beacon signals that can indicate a specific location, etc.

[0049] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.

[0050] Instructions 310 may be sent or received over network 322 via a network interface device (e.g., a network interface component included in communications component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 310 may be sent or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0051] Figure 4A is a process flow diagram of a text selection and modification process 400 for an AR system (such as but not limited to glasses 100 ) according to some examples. Figure 4B is an illustration of gestures according to some examples. Figure 4C , Figure 4D and Figure 4E is an illustration of an AR overlay 442 including a text scene object 444 with text 446, according to some examples. During the text selection and modification process 400, the AR system utilizes gesture recognition methods and DMVO methods to implement a user interface for text selection and modification during an AR experience provided by the AR system.

[0052] In operation 402, the AR system initiates the text selection and modification process 400 by invoking an AR text processing user interface application 426. The AR text processing user interface application 426 is connected to a scanning service component 432. The scanning service component 432 includes a gesture recognition service 430 that continuously recognizes gestures made by the user when using the AR system, and an object tracking service 428 that continuously tracks objects in the field of view of the AR system.

[0053] Object tracking service 428 scans, detects, and tracks objects in the real-world scene, including landmarks about portions of the user's upper body, arms, and hands in the real-world scene. In some examples, object tracking service 428 obtains information from one or more cameras of the AR system (such as Figure 1 The object tracking service 428 receives real-world scene video frame data of a real-world scene from the camera devices 114 and 116 of the user, and extracts features of objects including the upper body, arms, and hands of the user from the real-world scene video frame data. The object tracking service 428 generates current tracking data 436 based on the extracted features. The current tracking data 436 includes current object data of objects in the real-world scene and current skeleton model data 460, and the current skeleton model data 460 includes identification, location, and classification data of landmarks associated with the upper body, arms, and hands of the user. The object tracking service 428 transmits the current skeleton model data 460 to the gesture recognition service 430. In addition, the object tracking service 428 makes the current tracking data 436 available to the text selection and modification component 434.

[0054] The gesture recognition service 430 receives current skeletal model data 460 from the object tracking service 428 and compares the current skeletal model data 460 with previously generated gesture model data. The gesture recognition service 430 detects a detected gesture based on the comparison of the current skeletal model data 460 with the gesture model data, and generates current detected gesture data 462 based on the detected gesture. In another example, the gesture recognition service 430 determines the detected gesture based on classifying the current skeletal model data 460 using an artificial intelligence method and a gesture model previously generated using a machine learning method.

[0055] In operation 404, the AR system generates a text scene object 444 and adds the text scene object 444 to an AR overlay 442 provided by the AR system. For example, the AR device generates a text scene object 444 including one or more virtual text objects 468 based on the text 446. The one or more virtual text objects 468 correspond to respective characters of the text 446.

[0056] In operation 406, the AR system detects a text selection mode gesture made by the user based on the currently detected gesture data 462 received from the text selection and modification component 434, such as but not limited to Figure 4B In operation 408 , in response to detecting the text selection mode gesture 440 , the AR system provides a text selection and modification user interface as part of the text scene object 444 , and enters a text selection mode.

[0057] In operation 410 , the AR system detects a defined text selection gesture 448 (such as, but not limited to, a pinch gesture) made by a user using one of their hands as a selection hand 464 (such as, but not limited to, the user's left hand) based on currently detected gesture data 462 received from text selection and modification component 434 .

[0058] In operation 412, in response to detecting the text selection gesture 448, the AR system generates a selection line 450 based on the landmarks of the user's hand. For example, the AR system projects the selection line 450 from the landmarks of the user's hand making the text selection gesture 448, so that the selection line 450 is projected orthogonally toward the surface of the virtual text object 468 of the text scene object 444. The selection line acts as a virtual stylus for the user. In another example, the AR system determines the landmarks of the user's hand in the text selection gesture 448, such as but not limited to the midpoints of two fingers of the user's fingers performing the text selection gesture 448, based on the current skeletal model data 438 received from the text selection and modification component 434. The projected selection line 450 is the line collider that the user moves to select the text 446. When the user positions the selection line 450 within the virtual text object 468, the AR system detects a collision between the selection line 450 and the virtual text object 468 of the text scene object 444. The AR system determines the user interaction of the user with the virtual text object 468 based on the detected collision.

[0059] In operation 414, the AR system tracks the user's selection hand making the text selection gesture 448 as the user moves their selection hand to position the selection line 450 to a position at a text selection start point 452 in a virtual text object of a text scene object 444 based on the current skeletal model data 438 received from the text selection and modification component 434.

[0060] In operation 416 , the AR system detects a confirmation gesture 454 (such as but not limited to a fist gesture) made by the user using a confirmation hand 466 (such as but not limited to the user's right hand) based on currently detected gesture data 462 received by the AR system from text selection and modification component 434 .

[0061] In operation 418, the AR system sets a text selection start point 452 for selecting text from the text 446 at a position of an intersection of the selection line 450 and a start virtual text object in the virtual text object 468 of the text scene object 444. The text selection start point 452 is a start point of the text 446 of the text to be selected from the text 446.

[0062] In operation 420 , the AR system determines whether the text selection end point 456 has been set. Based on determining that the text selection end point 456 has not been set, the AR system repeats operations 414 , 416 , and 418 to set the text selection end point 456 .

[0063] In the repeated operation 414, the AR system tracks the user's selection hand making the text selection gesture 448 as the user moves their selection hand to position the selection line 450 at the location of the text selection end point 456 in the virtual text object 468 based on the current skeletal model data 438 received from the text selection and modification component 434.

[0064] In some examples, AR overlay 442 includes one or more user-selectable virtual directional keys, such as, but not limited to, directional key 470 , which the user can use to perform text selection once initial cursor placement is set.

[0065] In the repeated operation 416 , the AR system detects a subsequent confirmation gesture 454 made by the user using their confirmation hand 466 based on the currently detected gesture data 462 received by the AR system from the text selection and modification component 434 .

[0066] In the repeated operation 418, the AR system sets a text selection end point 456 for selecting text from the text 446 to the position of the intersection of the selection line 450 and the end virtual text object in the virtual text object 468. The text selection end point is the end point of the text 446 of the text to be selected from the text 446.

[0067] In the repeated operation 420, the AR system determines whether the text selection end point 456 has been set. Based on the determination that the text selection end point 456 has been set, the AR system selects the selected text 458 between the text selection start point 452 and the text selection end point 456 from the text 446. In operation 422, the AR system highlights the selected text 458 and performs a text operation on the selected text 458. The text operation can be a modification of the selected text 458, cutting the selected text 458, copying the selected text 458, changing the appearance of the selected text 458, changing the format of the selected text, etc.

[0068] In some examples, the AR system generates a virtual keyboard that is provided to the user in the AR overlay 442, such as, but not limited to: Figure 6C 458. The user can use the virtual keyboard 628 to edit the selected text 458 as described herein.

[0069] In some examples, the AR system generates a user-selectable text modification list that can be used to modify text, such as, but not limited to, cutting the selected text 458, copying the selected text 458, changing the appearance of the selected text 458, changing the format of the selected text, etc.

[0070] In operation 424 , the AR system exits the text selection mode and terminates the AR text processing user interface application 426 .

[0071] In some examples, the user makes a text selection mode exit gesture (such as, but not limited to, text selection mode gesture 440 ) to terminate the AR text processing user interface application 426 .

[0072] In some examples, the AR overlay 442 includes a user-selectable exit button (not shown) that, when selected by the user, terminates the AR text-processing user interface application 426 .

[0073] In some examples, the AR text processing user interface application 426 terminates after completing the operation on the selected text 458.

[0074] Figure 5A is a process flow diagram of a text selection and modification process 500 of an AR system (such as, but not limited to, glasses 100); Figure 5B is an illustration of a text selection mode gesture; and Figure 5A , Figure 5C , Figure 5D and Figure 5E is an illustration of an AR overlay 546 including a text scene object 542 with text 544, according to some examples. During the text selection and modification process 500, the AR system utilizes gesture recognition and head tracking methods to implement a user interface for text selection and modification during an AR experience provided by the AR system.

[0075] In operation 502, the AR system initiates the text selection and modification process 500 by invoking an AR text processing user interface application 524. The AR text processing user interface application 524 is connected to a scanning service component 530. The scanning service component 530 includes a gesture recognition service 528 that continuously recognizes gestures made by the user while using the AR system, an object tracking service 526 that continuously tracks objects in the field of view of the AR system, and a head tracking service 538 that continuously tracks the movement of the user's head while wearing the AR system.

[0076] The object tracking service 526 scans, detects, and tracks objects in the real-world scene, including landmarks about portions of the upper body, arms, and hands of the user in the real-world scene. In some examples, the object tracking service 526 receives real-world scene video frame data from one or more cameras of the AR system (such as, but not limited to, cameras 114 and 116), and extracts features of objects from the real-world scene video frame data, the objects including the upper body, arms, and hands of the user. The object tracking service 526 generates current tracking data 556 based on the extracted features. The current tracking data 556 includes current object data and current skeleton model data 554 of the object in the real-world scene, and the current skeleton model data 554 includes identification, location, and classification data of landmarks associated with the upper body, arms, and hands of the user. The object tracking service 526 transmits the current skeleton model data 554 to the gesture recognition service 528. In addition, the object tracking service 526 makes the current tracking data 556 available to the text selection and modification component 532.

[0077] The gesture recognition service 528 receives the current skeletal model data 554 from the object tracking service 526 and compares the current skeletal model data 554 with the previously generated gesture model data. The gesture recognition service 528 detects the detected gesture based on the comparison of the current skeletal model data 554 with the gesture model data, and generates the current detected gesture data 534 based on the detected gesture. In another example, the gesture recognition service 528 determines the detected gesture based on classifying the current skeletal model data 554 using an artificial intelligence method and a gesture model previously generated using a machine learning method.

[0078] Head Tracking Head tracking service 538 collects information from the AR system's IMU (such as but not limited to Figure 3 The IMU included in the positioning component 340 receives the head tracking data and makes the current head tracking data 536 available to the text selection and modification component 532.

[0079] During execution of the AR text processing user interface application 524, in operation 504, the AR system adds a text scene object 542 to the AR overlay 546. The text scene object 542 includes text 544 to be selected and modified by the user using the AR system. For example, the AR system generates a text scene object 542 including one or more virtual text objects 560 based on the text 544. The one or more virtual text objects 560 correspond to respective characters of the text 544.

[0080] In operation 506, the AR system detects a text selection mode gesture, such as but not limited to a start / stop text selection mode gesture 540, based on currently detected gesture data 534 received from text selection and modification component 532. In operation 508, in response to detecting the text selection mode gesture, the AR system provides a text selection and modification user interface as part of text scene object 542 and enters text selection mode.

[0081] In operation 510, the AR system projects a text selection icon 548 onto a potential text selection point in the text 544 of the text scene object 542 based on the current head tracking data generated by the head tracking service 538. For example, the AR system determines the direction in which the user is looking in the real world scene based on the current head tracking data 536. The AR system generates a ray originating from the user's eye position and projects the ray into the direction in which the user is looking. The AR system determines the user's focus point based on the position of the intersection of the ray with the text 544 of the text scene object 542. The AR system generates a text selection icon 548 at the location of the user's focus point in the text 544.

[0082] In operation 512 , the AR system detects a confirmation gesture, such as confirmation gesture 550 , made by a user using a confirmation hand 558 , such as the user's right hand, based on currently detected gesture data 534 received by the AR system from text selection and modification component 532 .

[0083] In operation 514, in response to detecting the confirmation gesture 550, the AR system sets a text selection starting point in the text 544 based on the current potential text selection point located at the location where the AR system currently projects the text selection icon 548 onto the text 544. The text selection starting point is a starting point in the text 544 of text to be selected from the text 544.

[0084] In operation 516, the AR system determines whether the text selection end point has been set. The AR system repeats operation 510 (projecting text selection icon 548 onto a potential text selection point), operation 512 (detecting a subsequent confirmation gesture made using the confirmation hand), and operation 514 (setting the text selection end point) to set the text selection end point based on determining that the text selection end point has not been set.

[0085] In the repeated operation 510 , the AR system projects a text selection icon 548 onto a potential text selection point in the text 544 of the text scene object 542 based on the current head tracking data generated by the head tracking service 538 .

[0086] In repeated operation 512 , the AR system detects a subsequent confirmation gesture 550 made by the user using the user's confirmation hand 558 based on currently detected gesture data 534 received by the AR system from text selection and modification component 532 .

[0087] In the repeated operation 514, in response to detecting the confirmation gesture 550, the AR system sets a text selection end point in the text 544 based on the subsequent current potential text selection point located at the location where the AR system currently projects the text selection icon 548 onto the text 544. The text selection end point is an end point in the text 544 of the text to be selected from the text 544.

[0088] In the repeated operation 516, the AR system determines whether a text selection end point has been selected. In operation 518, the AR system selects the selected text 552 and highlights the selected text 552 between the text selection start point and the text selection end point based on determining that the text selection end point has been set. In operation 520, the AR system performs a text operation on the selected text 552. The text operation can be a modification of the selected text 552, cutting the selected text 552, copying the selected text 552, changing the appearance of the selected text 458, changing the format of the selected text, etc. In operation 522, the AR system exits the text selection mode and terminates.

[0089] Fig. 6A is a process flow diagram of a virtual keyboard process 600 of an AR system (such as but not limited to glasses 100 ) according to some embodiments. Figure 6B is a user-selected depiction of physical surface 640 according to some embodiments. Figure 6C is an illustration of a virtual keyboard user interface 660 according to some implementations. During virtual keyboard processing 600, the AR system utilizes gesture recognition methods and DMVO methods to implement a virtual keyboard 628 of a virtual keyboard user interface 660 on a physical surface 640 for text selection and modification during an AR experience.

[0090] The AR system executes a virtual keyboard application 620 to provide the user with a virtual keyboard user interface 660. The virtual keyboard application 620 is connected to a scanning service component 626 of the AR system. The scanning service component 626 includes a gesture recognition service 624 that continuously recognizes gestures made by the user when using the AR system, an object tracking service 622 that continuously tracks objects in the field of view of the AR system, and a head tracking service 646 that continuously tracks the movement of the user's head when wearing the AR system.

[0091] The object tracking service 622 detects the object according to one or more cameras of the AR system (such as but not limited to Figure 1The camera devices 114 and 116 of the AR system receive real-world scene video frame data of the real-world scene from the perspective of the user of the AR system. Included in the real-world scene video frame data is tracking video frame data of objects in the real-world scene and parts of the user's body (including parts of the user's upper torso, arms, hands, and fingers). The tracking video frame data includes: video frame data of the movement of parts of the user's upper torso, arms, and hands when the user makes gestures or moves their hands and fingers to interact with objects in the AR overlay of the AR experience; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves their hands and fingers to interact with objects in the AR overlay of the AR experience; video frame data of the position of the user's upper torso, arms, hands, and fingers when the user makes gestures or moves their hands and fingers to interact with objects in the AR overlay of the AR experience; and object video frame data of objects in the real-world scene.

[0092] The object tracking service 622 scans, detects, and tracks objects in a real-world scene, including landmarks about portions of the upper body, arms, and hands of a user in the real-world scene. In some examples, the object tracking service 622 receives real-world scene video frame data from one or more cameras, and extracts features of objects including the upper body, arms, and hands of the user from the tracking video frame data included in the real-world scene video frame. The object tracking service 622 generates current tracking data based on the extracted features. The current tracking data includes current object data 656 and current skeleton model data 648 of the object in the real-world scene, and the current skeleton model data 648 includes identification, location, and classification data of landmarks associated with the upper body, arms, and hands of the user. The object tracking service 622 transmits the current skeleton model data 648 to the gesture recognition service 624. In addition, the object tracking service 622 makes the current tracking data available to the virtual keyboard application 620.

[0093] The gesture recognition service 624 receives the current skeletal model data 648 from the object tracking service 622 and compares the current skeletal model data 648 with the previously generated gesture model data. The gesture recognition service 624 detects the detected gesture based on the comparison of the current skeletal model data 648 with the gesture model data, and generates the current detected gesture data 650 based on the detected gesture. In another example, the gesture recognition service 624 determines the detected gesture based on classifying the current skeletal model data 648 using an artificial intelligence method and a gesture model previously generated using a machine learning method.

[0094] The head tracking service 646 receives head tracking data from the AR system's IMU and makes the current head tracking data 652 available to the virtual keyboard application 620 .

[0095] In some examples, tracking gesture recognition service 624, gesture recognition service 624, and head tracking service 646 operate continuously so that current skeletal model data 648, current detected gesture data 650, and current head tracking data 652 are available as needed to applications executing on the AR system.

[0096] During the AR experience, the user approaches a physical object 654 having a flat physical surface 640 on which a virtual keyboard 628 is projected. The user focuses on a text scene object 636 of an AR overlay 658 within the AR experience and physically touches the physical surface 640 with a text input hand 630 (such as, but not limited to, their left hand). The user makes an open keyboard gesture with a free hand 642 (such as, but not limited to, a right hand), such as, but not limited to, an open / close keyboard gesture 644. In operation 602, the AR system detects an open keyboard gesture based on currently detected gesture data 650 generated by gesture recognition service 624. In response to detecting the open keyboard gesture, the AR system invokes virtual keyboard application 620.

[0097] In some examples, the AR system determines the user's focus on the physical surface location based on the current head tracking data received from the scanning service component 626. The physical surface location is determined based on the physical object that the user is focusing on without causing the user to touch the physical surface. When the user is focusing on the surface that the user wants to select, the user uses a free hand to make an open keyboard gesture, such as but not limited to open / close keyboard gesture 644. The AR system detects open / close keyboard gesture 644 and determines the physical surface that the user is currently focusing on as the physical surface location onto which the virtual keyboard is to be virtually projected within the AR experience.

[0098] In some examples, the user makes a tap gesture with a finger on a physical surface to indicate the physical surface onto which the virtual keyboard is to be virtually projected within the AR experience. The AR system detects the tap motion and determines the location of the physical surface based on the current skeletal model data received from the scanning service component 626. When the AR system generates the virtual keyboard, the virtual keyboard is centered on the location determined by the user's finger.

[0099] In some examples, the AR system highlights the physical surface based on the user's attention to the physical surface. In some examples, the AR system highlights the physical surface based on a determination of a location of the physical surface determined from the user touching the physical surface.

[0100] In operation 604, the AR device detects the user's attention to the text scene object 636 based on the current head tracking data 652 provided by the head tracking service 646 and the position of the text scene object 636 in the AR overlay 658. In some examples, the AR device receives the current head tracking data 652 from the head tracking service 646. The AR system determines the direction the user is looking at in the real world scene based on the current head tracking data 536. The AR system generates a ray originating from the user's eye position and projects the ray in the direction the user is looking. The AR system detects the text scene object 636 based on the intersection of the ray and the text scene object 636.

[0101] In operation 606, the AR device determines a position in the user interface model that corresponds to the position of the physical surface 640 of the physical object 654 in the real world scene based on the current object data 656 received from the object tracking service 622. For example, the AR device determines the position of a landmark associated with the fingertip of the index finger of the text input hand 630 based on the current skeletal model data 648 received from the object tracking service 622. When the user is touching the physical surface 640, the position of the physical surface 640 is the position of the landmark associated with the fingertip of the index finger of the user's text input hand 630. The AR device generates a geometric mesh of a virtual surface in the user interface model that corresponds to the physical surface 640 based on the position of the physical surface 640 and the current object data 656 received from the object tracking service 622.

[0102] In operation 608, the AR device generates a virtual keyboard user interface 660 that includes a virtual keyboard 628 in an AR overlay 658 and is associated with a text scene object 636. The virtual keyboard 628 includes a plurality of virtual objects that constitute interactive virtual keys 632 of the virtual keyboard 628. The virtual keys 632 are virtual objects that represent geometric entities that have corresponding positions in the user interface model or have volumes that correspond to the volume of the space in the real world scene occupied by the virtual keyboard 628. The AR device positions the virtual keyboard 628 at a location in the user interface model that corresponds to the location of the physical surface 640 in the real world scene, and orients the virtual keyboard 628 so that a front surface of the virtual keyboard 628 corresponds to a geometric mesh of the virtual surface determined for the physical surface 640 of the physical object.

[0103] In operation 610, the AR device provides a virtual keyboard user interface 660 including a virtual keyboard 628 to the user in a display, and the virtual keyboard 628 appears to the user as if the virtual keyboard 628 is located on a physical surface 640 of a physical object 654. When the user interacts with the virtual keyboard 628, when the user selects one or more virtual keys 632 of the virtual keyboard 628, the user touches the physical surface.

[0104] In operation 612, the AR device determines a user's selection of one or more selected virtual keys. For example, the AR device receives current skeletal model data 648 from the object tracking service 622. The AR device detects landmarks associated with the user's text input hand 630 (e.g., the tip of an index finger) based on the current skeletal model data 648. The AR device generates a landmark collider 634 in the virtual keyboard user interface 660 based on the landmarks. To select a virtual key from the virtual keys 632, the user moves their text input hand 630 to move the landmark collider 634 within the virtual keyboard user interface 660. To move the landmark collider 634 on the virtual key 632 without selecting the virtual key, the user retracts their text input hand 630 away from the physical surface 640, and thereby retracts the virtual key 632 in the user interface model, thereby clearing the virtual key 632. To select a virtual key, the user positions their text input hand 630 over a virtual key and extends their text input hand 630 to "press" or "poke" the virtual key until the landmark collider 634 collides with the selected virtual key and the user physically touches the physical surface 640. The AR device determines the user's selection of one or more selected virtual keys based on detecting a collision between the landmark collider 634 and the one or more selected virtual keys.

[0105] In operation 614, the AR device provides the input text to the user in a text scene object 636. For example, each of the selected virtual keys in the virtual keys 632 is associated with a character. The AR device generates the input text data based on the selected virtual key and transmits the input text data to the text scene object. The text scene object receives the input text data and uses the optical engine of the AR system to use the display of the AR system to provide the input text data as input text to the user.

[0106] In operation 616, the AR device determines whether the user is making a close keyboard gesture, such as, but not limited to, an open / close keyboard gesture 644. For example, the AR device receives currently detected gesture data 650 from the gesture recognition service 624, and determines whether the gesture being made by the user is a close keyboard gesture based on the currently detected gesture data 650. In operation 618, the AR system closes the virtual keyboard user interface 660 and terminates the virtual keyboard application 620 based on determining that the user is making a close keyboard gesture. Based on determining that the user is not making a close keyboard gesture, the AR system continues to determine the user's selection of one or more virtual keys in operation 612.

[0107] In some examples, the characters of the input text data 638 provided to the user in the text scene object 636 are composed of virtual objects that the user interacts with using the landmark collider 634. For example, the AR device detects the user's selection of characters from the text scene object 636 based on detecting the collision between the landmark collider 634 and the selected characters. Therefore, the AR device can determine the user's selection or manipulation of the input text data by detecting the collision between the landmark collider 634 and the virtual object of the character. Text selection can include: the AR device detects that the user drags the landmark collider 634 at the location of the input text data 638 containing the character to be selected on the physical surface 640 using their text input hand 630 to select and highlight the text area. In some examples, the AR device provides a menu user interface to the user within the virtual keyboard user interface 660. The menu user interface contains selectable text manipulation options in the form of virtual objects, including copy, cut and paste. The AR device detects the user's selection of a text operation item based on detecting the collision between the landmark collider 634 and the virtual object of the selectable text manipulation option.

[0108] In some examples, an AR device of the AR system performs the functions of the gesture recognition service 624 and the head tracking service 646 by utilizing various APIs and system libraries.

[0109] Figure 7 700 is a block diagram illustrating a software architecture 704 that may be installed on any one or more of the devices described herein. The software architecture 704 is supported by hardware such as a machine 702 including a processor 720, a memory 726, and an I / O component 738. In this example, the software architecture 704 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 704 includes layers such as an operating system 712, a library 708, a framework 710, and an application 706. In operation, the application 706 invokes an API call 750 through the software stack and receives a message 752 in response to the API call 750.

[0110] The operating system 712 manages hardware resources and provides public services. The operating system 712 includes, for example, a kernel 714, services 716, and drivers 722. The kernel 714 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 714 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 716 can provide other public services to other software layers. Drivers 722 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 722 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0111] The library 708 provides a low-level common infrastructure used by the application 706. The library 708 may include a system library 718 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 708 may include an API library 724, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats such as Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 708 may also include various other libraries 728 to provide many other APIs to the application 706 .

[0112] The framework 710 provides a high-level common infrastructure used by the applications 706. For example, the framework 710 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 710 can provide a wide range of other APIs that can be used by the applications 706, some of which can be specific to a particular operating system or platform.

[0113] In some examples, applications 706 may include home applications 736, contact applications 730, browser applications 732, book reader applications 734, location applications 742, media applications 744, messaging applications 746, game applications 748, and a broad category of other applications such as third-party applications 740. Applications 706 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 706 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, third-party applications 740 (e.g., applications created by entities other than the vendor of a particular platform using ANDROID TM or IOS TMSoftware Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the Phone or another mobile operating system. In this example, third-party applications 740 can activate API calls 750 provided by the operating system 712 to facilitate the functions described herein.

[0114] Figure 8 830 is a block diagram showing a networked system 800 including details of the glasses 100 according to some examples. The networked system 800 includes the glasses 100, a client device 826, and a server system 832. The client device 826 may be a smart phone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 836 and / or a high-speed wireless connection 834. The client device 826 is connected to the server system 832 via a network 830. The network 830 may include any combination of wired and wireless connections. The server system 832 may be one or more computing devices that are part of a service or network computing system. The client device 826 and any elements of the server system 832 and the network 830 may be connected using a wireless network, such as a wireless network or a wireless network. Figure 7 and Figure 3 The details of the software architecture 704 or computing device 300 described in the embodiment of the present invention are implemented.

[0115] The glasses 100 include a data processor 802, a display 810, one or more cameras 808, and additional input / output elements 816. The input / output elements 816 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 802. Figure 7 and Figure 3 Examples of input / output elements 816 are further discussed. For example, input / output elements 816 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 810 are discussed in . In the specific examples described herein, display 810 includes displays for the left and right eyes of a user.

[0116] The data processor 802 includes an image processor 806 (eg, a video processor), a GPU & display driver 838, a tracking module 840, an interface 812, low power circuitry 804, and high speed circuitry 820. The components of the data processor 802 are interconnected by a bus 842.

[0117] The interface 812 refers to any source of user commands provided to the data processor 802. In one or more examples, the interface 812 is a physical button that, when pressed, sends a user input signal from the interface 812 to the low-power processor 814. The low-power processor 814 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 814 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 812 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from the camera 808. In other examples, the interface 812 can have a software component, or can be associated with a command received wirelessly from another source such as from the client device 826.

[0118] The image processor 806 includes circuitry for receiving signals from the camera 808 and processing those signals from the camera 808 into a format suitable for storage in the memory 824 or for transmission to the client device 826. In one or more examples, the image processor 806 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the camera 808, and volatile memory used by the microprocessor in operation.

[0119] The low power circuit system 804 includes a low power processor 814 and a low power wireless circuit system 818. These elements of the low power circuit system 804 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 814 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 814 can accept user input signals from the interface 812. The low power processor 814 can also be configured to receive input signals or command communications from a client device 826 via a low power wireless connection 836. The low power wireless circuit system 818 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 818. In other examples, other low power communication systems may be used.

[0120] High-speed circuit system 820 includes a high-speed processor 822, a memory 824, and a high-speed wireless circuit system 828. High-speed processor 822 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 802. High-speed processor 822 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 834 using high-speed wireless circuit system 828. In some examples, high-speed processor 822 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 7 The high-speed processor 822, which executes the software architecture of the data processor 802, manages data transmission with the high-speed wireless circuit system 828, in addition to any other duties. In some examples, the high-speed wireless circuit system 828 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 828 can implement other high-speed communication standards.

[0121] The memory 824 includes any storage device capable of storing camera data generated by the camera 808 and the image processor 806. Although the memory 824 is shown as being integrated with the high-speed circuitry 820, in other examples, the memory 824 may be a separate, independent element of the data processor 802. In some such examples, electrical wiring may provide a connection from the image processor 806 or the low-power processor 814 to the memory 824 through a chip including the high-speed processor 822. In other examples, the high-speed processor 822 may manage addressing of the memory 824 so that the low-power processor 814 will initiate the high-speed processor 822 whenever a read or write operation involving the memory 824 is desired.

[0122] The tracking module 840 estimates the pose of the glasses 100. For example, the tracking module 840 uses image data and associated inertial data from the camera 808 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking module 840 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative positioning and orientation relative to physical objects in the real-world scene environment. The tracking module 840 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 810.

[0123] The GPU & display driver 838 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 810 when the glasses 100 are operating in a traditional augmented reality mode. In this mode, the GPU & display driver 838 generates updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects the changes in the user's position and orientation relative to physical objects in the user's real-world scene environment.

[0124] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 826 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 706, such as the messaging application 746.

[0125] Fig. 9 is a block diagram illustrating an example messaging system 900 for exchanging data (e.g., messages and associated content) over a network. The messaging system 900 includes multiple instances of a client device 826 that host several applications including a messaging client 902 and other applications 904. The messaging client 902 is communicatively coupled to other instances of the messaging client 902 (e.g., hosted on respective other client devices 826), a messaging server system 906, and a third-party server 908 via a network 830 (e.g., the Internet). The messaging client 902 may also communicate with the locally hosted application 904 using an application program interface (API).

[0126] The messaging clients 902 are able to communicate and exchange data with other messaging clients 902 and with a messaging server system 906 via the network 830. The data exchanged between the messaging clients 902 and between the messaging clients 902 and the messaging server system 906 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0127] The messaging server system 906 provides server-side functionality to a particular messaging client 902 via the network 830. Although some functionality of the messaging system 900 is described herein as being performed by the messaging client 902 or by the messaging server system 906, it may be a design choice to locate some functionality within the messaging client 902 or within the messaging server system 906. For example, it may be technically preferable to initially deploy some technologies and functionality within the messaging server system 906, but then migrate the technologies and functionality to the messaging client 902 where the client device 826 has sufficient processing power.

[0128] The messaging server system 906 supports various services and operations provided to the messaging client 902. Such operations include sending data to the messaging client 902, receiving data from the messaging client 902, and processing data generated by the messaging client 902. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 900 is activated and controlled by functions available via the user interface (UI) of the messaging client 902.

[0129] Turning now specifically to the messaging server system 906, an application program interface (API) server 910 is coupled to and provides a programming interface to an application server 914. The application server 914 is communicatively coupled to a database server 916, which facilitates access to a database 920 that stores data associated with messages processed by the application server 914. Similarly, a web server 924 is coupled to and provides a web-based interface to the application server 914. To this end, the web server 924 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0130] The application program interface (API) server 910 receives and sends message data (e.g., commands and message payloads) between the client device 826 and the application server 914. Specifically, the application program interface (API) server 910 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 902 to activate the functions of the application server 914. The application program interface (API) server 910 exposes various functions supported by the application server 914, including: account registration; login functionality; sending messages from a particular messaging client 902 to another messaging client 902 via the application server 914, sending media files (e.g., images or videos) from the messaging client 902 to the messaging server 912, and possible access by another messaging client 902; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 826; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 902).

[0131] The application server 914 hosts several server applications and subsystems, including, for example, a messaging server 912, an image processing server 918, and a social network server 922. The messaging server 912 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 902. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 902. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 912.

[0132] The application server 914 also includes an image processing server 918 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 912.

[0133] The social network server 922 supports various social networking functions and services and makes these functions and services available to the messaging server 912. To this end, the social network server 922 maintains and accesses an entity graph within the database 920. Examples of functions and services supported by the social network server 922 include identifying other users in the messaging system 900 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0134] The messaging client 902 may notify the user of the client device 826 or other users associated with such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 902 may provide notifications related to current or recent use of a game by one or more members of a user group to participants in a conversation (e.g., a chat session) in the messaging client 902. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0135] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0136] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0137] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0138] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") is to be understood as containing a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a particular manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be realized by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be realized, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by a temporary configuration (e.g., by software) or a permanent configuration to perform one or more processors of the related operation. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, not only resides in a single machine, but also deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0139] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0140] "Machine storage media" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media."

[0141] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0142] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0143] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method comprising: generating, by one or more processors of an augmented reality (AR) system, a text scene object including one or more virtual text objects based on the text; detecting, by the one or more processors, using one or more cameras of the AR system, a text selection gesture made by a user of the AR system; generating, by the one or more processors, a selection line based on landmarks of the user's hand; detecting, by the one or more processors using the one or more cameras, a confirmation gesture made by the user; In response to detecting the confirmation gesture, setting, by the one or more processors, a text selection starting point at an intersection of the selection line and a starting virtual text object among the one or more virtual text objects; detecting, by the one or more processors, using the one or more cameras, a subsequent confirmation gesture by the user; in response to detecting the subsequent confirmation gesture, setting, by the one or more processors, a text selection end point at an intersection of the selection line and an end virtual text object among the one or more virtual text objects; as well as The selected text is selected, by the one or more processors, from the text based on the text selection start point and the text selection end point.

2. The computer-implemented method of claim 1 , further comprising: detecting, by the one or more processors, using the one or more cameras, a text selection mode gesture made by the user; as well as In response to detecting the text selection mode gesture, a text selection and modification user interface is provided by the one or more processors as part of the text scene object.

3. The computer-implemented method of claim 1 , wherein: The confirmation gesture and the subsequent confirmation gesture are the same gesture.

4. The computer-implemented method of claim 1 , wherein: The landmark of the selection hand is a midpoint of two of the user's fingers performing the text selection gesture.

5. The computer-implemented method of claim 1 , wherein: The selection line is a virtual stylus.

6. The computer-implemented method of claim 1 , wherein: The AR system includes a head-mounted device.

7. The computer-implemented method of claim 1 , wherein: The text selection gesture is made by the user using the user's selecting hand, and The landmarks of the selection line are the landmarks of the selection hand.

8. The computer-implemented method of claim 7, wherein: The confirmation gesture is made by the user using the user's confirmation hand.

9. A computing device comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: generating a text scene object including one or more virtual text objects based on the text; detecting, using one or more cameras of an AR system, a text selection gesture made by a user of the AR system using a selection hand of the user; generating a selection line based on the landmarks of the selection hand; detecting, using the one or more cameras, a confirmation gesture made by the user using the user's confirmation hand; In response to detecting the confirmation gesture, setting a text selection start point at an intersection of the selection line and a start virtual text object among the one or more virtual text objects; detecting, using the one or more cameras, a subsequent confirmation gesture made by the user; In response to detecting the subsequent confirmation gesture, setting a text selection end point at an intersection of the selection line and an end virtual text object among the one or more virtual text objects; and The selected text is selected from the text based on the text selection start point and the text selection end point.

10. The computing device according to claim 9, wherein: The instructions further cause the computing device to: detecting, using the one or more cameras, a text selection mode gesture by the user; and In response to detecting the text selection mode gesture, a text selection and modification user interface is provided as part of the text scene object.

11. The computing device according to claim 9, wherein: The confirmation gesture and the subsequent confirmation gesture are the same gesture.

12. The computing device according to claim 9, wherein: The landmark of the selection hand is a midpoint of two of the user's fingers performing the text selection gesture.

13. The computing device of claim 9, wherein: The selection line is a virtual stylus.

14. The computing device of claim 9, wherein: The AR system includes a head-mounted device.

15. The computing device of claim 9, wherein: The text selection gesture is made by the user using the user's selecting hand, and The landmarks of the selection line are the landmarks of the selection hand.

16. The computing device of claim 15, wherein: The confirmation gesture is made by the user using the user's confirmation hand.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: generating, by one or more processors of the AR device, a text scene object including one or more virtual text objects based on the text; detecting, by the one or more processors using one or more cameras of the AR system, a text selection gesture made by a user of the AR system using a selection hand of the user; generating, by the one or more processors, a selection line based on landmarks of the selection hand; detecting, by the one or more processors using the one or more cameras, a confirmation gesture made by the user using the user's confirmation hand; In response to detecting the confirmation gesture, setting, by the one or more processors, a text selection starting point at an intersection of the selection line and a starting virtual text object among the one or more virtual text objects; detecting, by the one or more processors using the one or more cameras, a subsequent confirmation gesture by the user; in response to detecting the subsequent confirmation gesture, setting, by the one or more processors, a text selection end point at an intersection of the selection line and an end virtual text object among the one or more virtual text objects; as well as The selected text is selected, by the one or more processors, from the text based on the text selection start point and the text selection end point.

18. The non-transitory computer readable storage medium of claim 17, wherein: The instructions also cause the computer to: detecting, using the one or more cameras, a text selection mode gesture by the user; and In response to detecting the text selection mode gesture, a text selection and modification user interface is provided by the one or more processors as part of the text scene object.

19. The non-transitory computer-readable storage medium of claim 17, wherein: The confirmation gesture and the subsequent confirmation gesture are the same gesture.

20. The non-transitory computer readable storage medium of claim 17, wherein: The landmark of the selection hand is a midpoint of two of the user's fingers performing the text selection gesture.

21. The non-transitory computer readable storage medium of claim 17, wherein: The selection line is a virtual stylus.

22. The non-transitory computer readable storage medium of claim 17, wherein: The AR system includes a head-mounted device.

23. The non-transitory computer readable storage medium of claim 17, wherein: The text selection gesture is made by the user using the user's selecting hand, and The landmarks of the selection line are the landmarks of the selection hand.

24. The non-transitory computer readable storage medium of claim 23, wherein: The confirmation gesture is made by the user using the user's confirmation hand.

Images