Computer-implemented method, computing device and non-transitory computer-readable storage medium

Abstract

The invention relates to a computer-implemented method, a computing device and a non-transitory computer-readable storage medium. The computer-implemented method includes capturing, by a first mobile device, an image including a fiducial marker displayed on a second mobile device; determining a relative pose between the first mobile device and the second mobile device by generating a pose transformation of the fiducial marker displayed by the second mobile device; receiving information specifying an update to the gesture of the second mobile device; and processing the received information specifying an update to the gesture of the second mobile device as a user input to the first mobile device.

Description

[0001] This application is a divisional application of the invention patent application filed on July 18, 2023, with application number 202380054612.6 (international phase application number PCT / US2023 / 070415) and entitled "Secure peer-to-peer connection between mobile devices".

[0002] Relevant application data

[0003] This application claims priority to U.S. Patent Application No. 17 / 813,901, filed July 20, 2022, the contents of which are incorporated herein by reference as expressly stated. Technical Field

[0004] This disclosure relates generally to communication between mobile devices, and more specifically to establishing secure peer-to-peer connections between mobile devices. Background Technology

[0005] For non-technical users, establishing a secure communication link between nearby mobile devices often involves a cumbersome setup process. Some solutions, such as Apple's AirDrop, use Bluetooth broadcasting to identify mobile devices in the area. If a user wants to share a file, a list of potential recipients in the area is displayed.

[0006] Using Bluetooth broadcasting and associated information exchange poses potential security risks and requires users to specifically enable or disable file sharing to allow all devices in the area, disallow devices in the area, or allow compatible devices of contacts in the area to make initial connection and sharing requests. Furthermore, proprietary implementations such as AirDrop have limited use outside their ecosystem.

[0007] Additionally, when pairing devices via Bluetooth broadcast, a list of available devices is initially presented to the user for manual selection. This can be particularly challenging when there are multiple devices of the same brand and model. Attached Figure Description

[0008] To facilitate identification of any discussion of a particular element or action, one or more of the highest significant digits in the figure references indicate the figure number in which the element was first introduced.

[0009] Figure 1 These are stereoscopic images based on some examples of head-mounted devices.

[0010] Figure 2 The following are examples. Figure 1 Another view of the head-mounted device.

[0011] Figure 3 This is shown based on some examples, including Figure 1 A block diagram showing the details of the networked system for the head-mounted device.

[0012] Figure 4 This is a basic process flowchart illustrating the establishment of a peer-to-peer Bluetooth communication link based on some examples.

[0013] Figure 5 This is a flowchart illustrating a part of a method for establishing a peer-to-peer Bluetooth communication link, based on some examples.

[0014] Figure 6 This is a flowchart illustrating a part of a method for establishing a peer-to-peer Bluetooth communication link, based on some examples.

[0015] Figure 7A It is a view of the environment through the user's glasses, based on some examples, where the first mobile device is displaying a QR code.

[0016] Figure 7B These are views of the environment seen through a user's glasses, based on some examples, where a first mobile device is being used as an input device for the glasses.

[0017] Figure 8 This is a flowchart illustrating a portion of a method for establishing a peer-to-peer Bluetooth communication link and aligning coordinate systems, based on some examples.

[0018] Figure 9 It is a graphical representation of a networked environment in which the present disclosure can be deployed, based on some examples.

[0019] Figure 10 This is a block diagram illustrating a software architecture within which the present disclosure can be implemented, based on some examples.

[0020] Figure 11 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0021] In some examples, a secure peer-to-peer connection between mobile devices is established by displaying a QR code on the display of a first mobile device. The second mobile device scans the QR code and then establishes a connection with the first mobile device based on the information encoded in the QR code. In some cases, the first mobile device is a smartphone, and the second mobile device is a head-mounted augmented reality (AR) device.

[0022] A QR code is a machine-readable optical tag that can contain information about the item it is attached to. QR codes are widely used to share small amounts of data between mobile devices. For example, some messaging applications use QR codes to quickly allow users who display the QR code to be added as "friends" by the user who scanned the mobile device. QR codes are often also encoded with a URL, which allows mobile devices to retrieve additional information from remote servers.

[0023] As disclosed herein, the QR code displayed on the second device includes Bluetooth Low Energy (BLE) address information, which can be used to establish a BLE communication link between the two devices in an ad hoc connection without any additional setup. In traditional Bluetooth connection establishment, the user of the mobile device initially selects a device from a list of available devices and thus designates it as trusted. Then, in a process called bonding, the static address and other information of the selected Bluetooth device are stored in the mobile device's memory. Bonding involves exchanging long-term keys after pairing occurs and storing these keys for later use. Bonding creates a persistent, secure relationship between the two devices. Bonding facilitates the re-establishment of connections between trusted devices because the trusted device has been designated as such.

[0024] Using the scanning of a QR code displayed on one device to pair two devices avoids the requirement that a binding has already occurred, as the user will likely only scan the QR code on the mobile device they intend to connect to (i.e., a device trusted by the user). The Bluetooth address used can be dynamic, as it does not need to rely on the address of the paired device previously stored in memory and therefore required to be static to allow connection. In some cases, a "confirm connection" user interface prompt could be provided on both devices to prevent accidental scanning, but this is unlikely in any case.

[0025] In some examples, a computer-implemented method is provided, executed by a first mobile device, for establishing a communication link between the first mobile device and a second mobile device. The method includes: capturing an image of machine-readable code displayed on the second mobile device by the first mobile device; decoding communication setup information from the captured image of the machine-readable code by the first mobile device; and establishing a communication link with the second mobile device using the communication setup information. The communication setup information can be generated dynamically.

[0026] The computer-implemented method may further include: sending at least a portion of the broadcast communication settings information message, and / or scanning for at least a portion of the broadcast communication settings information message from the second mobile device.

[0027] In some examples, the method may include: determining a transformation of a reference marker displayed by a second mobile device; receiving information specifying the pose of the second mobile device; using the transformation and the information specifying the pose of the second mobile device to determine the relative pose between the first and second mobile devices; and aligning coordinate systems between the first and second mobile devices. The machine-readable code may be the reference marker. The information specifying the pose of the second mobile device may be encoded in the machine-readable code.

[0028] The method may further include: receiving updates to information about the posture of a designated second mobile device via a communication link; and performing user interface actions on a first mobile device based on the received updates to the posture of the designated second mobile device.

[0029] In some examples, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed by a first mobile device, cause the first mobile device to perform operations for establishing a communication link between the first mobile device and a second mobile device, in accordance with any of the methods and limitations described above, including but not limited to: capturing an image of machine-readable code displayed on the second mobile device by the first mobile device; decoding communication settings information from the captured image of the machine-readable code by the first mobile device; and establishing a communication link with the second mobile device using the communication settings information.

[0030] In some examples, a computing device is provided, comprising a processor and a memory storing instructions that, when executed by the processor, configure the device to perform operations for establishing a communication link between a first mobile device and a second mobile device, according to any of the methods and limitations described above, including but not limited to: capturing an image of machine-readable code displayed on the second mobile device by the first mobile device; decoding communication setup information from the captured image of the machine-readable code by the first mobile device; and establishing a communication link with the second mobile device using the communication setup information.

[0031] Other technical features may be readily apparent to those skilled in the art based on the accompanying drawings, description, and claims.

[0032] Figure 1This is a stereoscopic view of a head-mounted AR device (e.g., glasses 100) based on some examples. Glasses 100 may include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, frame 102 includes a first optical element holder or left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or right optical element holder 106 connected by a bridging portion 112. A first optical element or left optical element 108 and a second optical element or right optical element 110 may be disposed within the left optical element holder 104 and right optical element holder 106, respectively. The right optical element 110 and left optical element 108 may be a lens, a display, a display assembly, or a combination thereof. Any suitable display assembly may be disposed in glasses 100.

[0033] Frame 102 further includes a left arm or left temple 122 and a right arm or right temple 124. In some examples, frame 102 may be formed from a single piece of material to have a monolithic or integrated construction.

[0034] The eyeglasses 100 may include a computing device such as a computer 120, which may be of any suitable type for carrying by the frame 102, and in one or more examples, the computing device may have a suitable size and shape to be partially housed in one of the temple pieces 122 or 124. The computer 120 may include one or more processors, as well as memory, wireless communication circuitry, and a power supply. As discussed below, the computer 120 includes low-power circuitry, high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated in different ways. Additional details of the aspects of the computer 120 may be implemented as shown in the data processor 302 discussed below.

[0035] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple 122 and electrically coupled to the computer 120 disposed in the right temple 124. The glasses 100 may include a connector or port (not shown) for charging the battery 118, a wireless receiver, a transmitter or transceiver (not shown), or a combination of such devices.

[0036] The glasses 100 include a first camera device or left camera device 114 and a second camera device or right camera device 116. Although two camera devices are depicted, other examples contemplate the use of a single or additional (i.e., more than two) camera devices. In one or more examples, in addition to the left camera device 114 and the right camera device 116, the glasses 100 also includes any number of input sensors or other input / output devices. Such sensors or input / output devices may additionally include biometric sensors, positioning sensors, motion sensors, etc.

[0037] In some examples, the left camera 114 and the right camera 116 provide video frame data for the glasses 100 to use to extract 3D information from the real-world scene.

[0038] The glasses 100 may also include a touchpad 126, which is mounted to or integrated with one or both of the left temple 122 and the right temple 124. The touchpad 126 is generally arranged vertically, and in some examples is approximately parallel to the user's temple. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than that. Additional user input can be provided via one or more buttons 128, which, in the illustrated example, are located on the outer upper edges of the left optics retainer 104 and the right optics retainer 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from the user of the glasses 100.

[0039] Figure 2 The glasses 100 are shown from the user's perspective. For clarity, Figure 1 Several components shown have been omitted. For example... Figure 1 As described, Figure 2 The glasses 100 shown include a left optical element 108 and a right optical element 110, which are respectively fixed in a left optical element holder 104 and a right optical element holder 106.

[0040] The glasses 100 include: a forward optics 202, which includes a right projector 204 and a right near-eye display 206; and a forward optics 210, which includes a left projector 212 and a left near-eye display 216.

[0041] In some examples, the near-eye display is a waveguide. The waveguide includes reflective or diffractive structures (e.g., gratings and / or optical elements such as mirrors, lenses, or prisms). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the user's right eye to provide an image superimposed on or within the right optical element 110, representing a view of the real world seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the user's left eye to provide an image superimposed on or within the left optical element 108, representing a view of the real world seen by the user. A combination of the GPU, forward optics 202, left optical element 108, and right optical element 110 provides the optical engine for glasses 100. Glasses 100 uses the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of glasses 100.

[0042] However, it should be understood that other display technologies or configurations can be used within the optical engine to display images to the user in their field of view. For example, instead of the projector 204 and the waveguide, an LCD, LED, or other display panel or surface can be provided.

[0043] In use, the user of glasses 100 will be presented with information, content, and various 3D user interfaces on a near-eye display. As described in more detail herein, the user can then use touchpad 126 and / or buttons 128, associated devices (e.g., Figure 3 The user device (328) shown interacts with the glasses 100 via voice input or touch input and / or hand movements, positioning and location detected by the glasses 100.

[0044] Figure 3 This is a block diagram illustrating details of a networking system 300 including glasses 100 according to some examples. The networking system 300 includes glasses 100, user equipment 328, and server system 332. User equipment 328 may be a smartphone, tablet computer, tablet phone, laptop computer, access point, or any other such device capable of connecting to glasses 100 using low-power wireless connection 336 and / or high-speed wireless connection 334. User equipment 328 is connected to server system 332 via network 330. Network 330 may include any combination of wired and wireless connections. Server system 332 may be one or more computing devices as part of a service or network computing system. (The last sentence appears to be incomplete and possibly refers to separate components.) Figure 10 and Figure 11 The details of the software architecture 1004 or machine 1100 described herein are used to implement any elements of the user device 328, server system 332, and network 330.

[0045] The glasses 100 include a data processor 302, a display 310, one or more camera devices 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, additional sensors, or an additional display element integrated with the data processor 302. (About...) Figure 10 and Figure 11 Examples of input / output element 316 are further discussed. For example, input / output element 316 may include any I / O component 1106, which may include output component 1128, moving component 1136, etc. Figure 2 An example of display 310 is discussed herein. In the specific example described herein, display 310 includes displays for the user's left and right eyes.

[0046] The data processor 302 includes an image processor 306 (e.g., a video processor), a GPU and a display driver 338, a tracking module 340, an interface 312, a low-power circuit system 304, and a high-speed circuit system 320. The components of the data processor 302 are interconnected via a bus 342.

[0047] Interface 312 refers to any source of user commands provided to data processor 302. In one or more examples, interface 312 is a physical button that, when pressed, sends a user input signal from interface 312 to low-power processor 314. Low-power processor 314 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. Low-power processor 314 can process pressing such a button for a first time period as a request to capture video data while the button is pressed and to stop video capture when the button is released, wherein the video captured while the button is pressed is stored as a single video file. Alternatively, pressing the button for a longer time period can capture a still image. In some examples, interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera device 308. In other examples, interface 312 can have software components or can be associated with commands received wirelessly from another source, such as user equipment 328.

[0048] Image processor 306 includes circuitry for receiving signals from camera device 308 and processing those signals into a format suitable for storage in memory 324 or for transmission to user equipment 328. In one or more examples, image processor 306 (e.g., video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera device 308, and volatile memory used by the microprocessor in operation.

[0049] Low-power circuit system 304 includes a low-power processor 314 and a low-power wireless circuit system 318. These elements of low-power circuit system 304 can be implemented as separate components or as part of a single-chip system on a single IC. Low-power processor 314 includes logic for managing other elements of glasses 100. As described above, for example, low-power processor 314 can accept user input signals from interface 312. Low-power processor 314 can also be configured to receive input signals or command communication from user equipment 328 via low-power wireless connection 336. Low-power wireless circuit system 318 includes circuit elements for implementing a low-power wireless communication system. Bluetooth™ Smart (also known as Bluetooth™ Low Energy) is a standard implementation of the low-power wireless communication system that can be used to implement low-power wireless circuit system 318. In other examples, other low-power communication systems may be used.

[0050] The high-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. The high-speed processor 322 can be any processor capable of managing high-speed communication and operation of any general-purpose computing system used by the data processor 302. The high-speed processor 322 includes processing resources used by the high-speed wireless circuit system 326 to manage high-speed data transmission over the high-speed wireless connection 334. In some examples, the high-speed processor 322 executes an operating system such as LINUX or other operating systems. Figure 10 The operating system 1012 is one of other such operating systems. In addition to any other duties, a high-speed processor 322, using the software architecture of the execution data processor 302, manages data transmission with the high-speed wireless circuit system 326. In some examples, the high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 326 may implement other high-speed communication standards.

[0051] Memory 324 includes any storage device capable of storing camera data generated by camera device 308 and image processor 306. While memory 324 is shown as integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, independent element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 via a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 such that low-power processor 314 will bootstrap high-speed processor 322 whenever a read or write operation involving memory 324 is desired.

[0052] Tracking module 340 estimates the position and orientation (“pose”) of glasses 100. For example, tracking module 340 uses image data and corresponding inertial data from camera device 308 and position unit 1140, along with GPS data, to track and locate the glasses 100 and determine its pose relative to a reference frame (e.g., the real-world environment). Tracking module 340 continuously collects and uses updated sensor data describing the motion of glasses 100 to determine an updated three-dimensional pose of glasses 100, indicating changes in its relative position and orientation to physical objects in the real-world environment. Tracking module 340 allows glasses 100 to visually position virtual objects relative to physical objects within the user's field of vision via display 310.

[0053] The GPU and display driver 338 can use the pose of the glasses 100 to generate frames of virtual content or other content to be displayed on the display 310 when the glasses 100 is running in a regular augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated 3D pose of the glasses 100, which reflects changes in the user's position and orientation of physical objects in the user's real-world environment.

[0054] One or more functions or operations described herein can also be performed in an application residing on glasses 100, user device 328, or a remote server. For example, one or more functions or operations described herein can be performed by application 1006, such as messaging application 1046.

[0055] Figure 4This is a basic flowchart illustrating the establishment of a peer-to-peer Bluetooth communication link according to some examples. As shown in the diagram, a QR code 406 is displayed by a first mobile device 402. The QR code 406 has Bluetooth address information 408 encoded therein, which has been dynamically generated. The QR code is scanned and decoded by a second mobile device 404. Using the address information 408, the second mobile device 404 initiates a Bluetooth communication link 410 with the first mobile device 402. The Bluetooth communication link 410 enables the exchange of data 412 between the two devices. See below for further details. Figure 5 Further details of the method are discussed. The first mobile device 402, which displays the QR code, typically assumes a central role in the Bluetooth session, while the second mobile device 404 assumes a peripheral role, although this is not mandatory.

[0056] Address information 408 includes a dynamically generated BLE service / feature UUID (Universally Unique Identifier) ​​used to establish a communication link. Therefore, address information 408 is unique and explicitly defined in each application running on each mobile device. Normally, a conflict would occur if two or more people in the same area were wearing glasses 100 and both were attempting to connect their glasses 100 to their user devices 328. The current method avoids this problem, allowing all users in the same area to have a unique and secure communication link between their own devices.

[0057] Figure 5 This is a flowchart 500 illustrating a portion of a method for establishing a peer-to-peer Bluetooth communication link performed on a first mobile device 402 according to some examples. For illustrative purposes, the operations of flowchart 500 are described herein as occurring serially or linearly. However, multiple operations of flowchart 500 may occur in parallel. Furthermore, the operations of flowchart 500 do not need to be performed in the order shown, and / or one or more blocks of flowchart 500 need not be performed and / or may be replaced by other operations.

[0058] The method begins at operation 502, where the first mobile device 402 receives user input initiating a Bluetooth communication link with another device. In response, in operation 504, the first mobile device 402 generates a random UUID as a BLE service / feature ID. In operation 506, the first mobile device 402 generates a QR code representing the UUID and displays the QR code in operation 508. The first mobile device 402 then begins scanning for UUIDs to be broadcast in messages sent by the second mobile device 404 using its Bluetooth transceiver.

[0059] If no UUID is detected in operation 512, the method returns to operation 510, and the first mobile device 402 continues scanning for messages containing at least some UUIDs from the broadcast UUIDs sent by the second mobile device 404. If a UUID is detected in operation 512, the first mobile device 402 responds to the second mobile device 404 and establishes a secure Bluetooth communication link in operation 514. Then, in operation 516, the first mobile device 402 and the second mobile device 404 exchange data.

[0060] Figure 6 This is a flowchart 600 illustrating a method for establishing a peer-to-peer Bluetooth communication link according to some examples. For illustrative purposes, the operations of flowchart 600 are described herein as occurring serially or linearly. However, multiple operations of flowchart 600 may occur in parallel. Furthermore, the operations of flowchart 600 do not need to be performed in the order shown, and / or one or more blocks of flowchart 600 need not be performed and / or may be replaced by other operations.

[0061] The method begins at operation 602, where the second mobile device 404 receives user input initiating a Bluetooth communication link with another device. In response, at operation 604, the second mobile device 404 begins scanning for video feeds from a camera device within the second mobile device 404.

[0062] If no QR code is detected in operation 606, the method returns to operation 604, and the second mobile device 404 continues scanning for QR codes. If a QR code is detected in operation 606, the second mobile device 404 decodes the QR code in operation 608 to extract the UUID reflected therein. In operation 610, the second mobile device 404 uses its Bluetooth transceiver to send a message broadcasting at least some of the decoded UUIDs. In operation 612, in response to the message, the second mobile device 404 receives an acknowledgment of transmission from the first mobile device 402, establishes a Bluetooth communication link in operation 614, and begins exchanging data in operation 616.

[0063] It should be understood that, instead of sending the UUID in operations 610 and 612, the second mobile device 404 can perform this in flowchart 600. Figure 5 Operations 510 and 512 (scanning for UUID) in flowchart 500 are replaced by operations 510 and 512 (scanning for UUID). Instead of scanning for UUID in operations 510 and 512, the first mobile device 402 can be as shown in flowchart 500. Figure 6 Operations 610 and 612 in the code send the UUID.

[0064] As described above, in flowcharts 500 and 600, the first mobile device 402, which shows the QR, typically assumes a central role in the Bluetooth session, while the second mobile device 404 assumes a peripheral role, although this is not necessary.

[0065] Depending on the application, additional mobile devices can scan the QR code 406 displayed by the first mobile device 402, which acts as the central device, to establish a communication link to the first mobile device 402 as peripheral devices. Alternatively, multiple mobile devices can be connected together in a BLE mesh topology.

[0066] Figure 7A This is based on an example view of the environment 702 through a user's glasses 100, where a first mobile device 402 is displaying a QR code 704. In this case, in addition to the first mobile device 402 and glasses 100 (as a second mobile device), Figure 5 and Figure 6 In addition to this method, QR code 704 is also used to align the coordinate systems of two devices.

[0067] Both the first mobile device 402 and the glasses 100 perform visual inertial odometry to determine and track their respective positions and orientations (“postures”) within the environment 702. The relative posture between the glasses 100 and the first mobile device 402 can be determined by having the glasses 100 capture an image of a reference marker. In this case, the QR code also serves as a reference marker, in addition to providing information for establishing a communication link. Since the shape and correct orientation of the reference marker are known, the glasses 100 can determine the posture transformation between the captured image of the reference marker and the actual reference marker.

[0068] In some examples, the pose of the first mobile device 402 can be encoded in the QR code 704, allowing the glasses 100 to capture an image of the QR code 704, determine the transformations of the QR code 704 image, and determine the relative pose between the first mobile device 402 and the glasses 100. In other examples, the pose of the first mobile device 402 is reported to the glasses 100 via the Bluetooth communication link 410 after it has been established, while the QR code continues to be displayed. The glasses 100 can then capture the image of the QR code and the current pose of the first mobile device 402 almost simultaneously. The latter method is less sensitive to movement of the first mobile device 402, which may occur after the QR code, including the pose of the mobile device, has been generated but before it has been captured by the glasses 100.

[0069] The determined transformation will provide the relative posture between the first mobile device 402 and the glasses 100. Either or both of the coordinate systems of the first mobile device 402 and the glasses 100 can then be adjusted so that the postures of the two devices are represented in the same coordinate system. The first mobile device 402 then reports its posture and touch input to the glasses 100 via Bluetooth communication link 410.

[0070] Figure 7A The image also shows a text prompt 706 displayed to the user by the glasses 100, which instructs the user to keep the phone in front of the glasses 100 so that the QR code 406 can be imaged and decoded.

[0071] Figure 7B This is a view of the environment 702 through a user's glasses 100, based on some examples, where a first mobile device 402 is being used as an input device for the glasses 100. Figure 7B A Bluetooth communication link has been established, and the coordinate systems of the two devices have been aligned as described above. The first mobile device 402 is reporting its posture and any received user touch input to the glasses 100 via the Bluetooth communication link 410.

[0072] As shown, the first mobile device 402 is displaying user interface elements 708, including, for example, a directional touchpad that can be used to move a cursor 714 displayed by the glasses 100. A user interface window 710 displayed by the glasses 100 is also shown.

[0073] By combining touch input and gestures using the first mobile device 402, a user can interact with augmented reality elements displayed by the glasses, such as AR element 712 and window 710. For example, using an directional touchpad shown on the touchscreen of the first mobile device 402, a user can place cursor 714 over window 710. A button shown in the window can then be activated with a single touch on the touchscreen of the first mobile device 402, or the window can be moved within the field of view of the glasses 100 by pressing and holding on the touchscreen and then pointing the first mobile device 402 to the desired location of window 710.

[0074] Therefore, the first mobile device 402 serves as a six-degrees-of-freedom controller with touch input for the glasses 100. This alleviates some of the challenges of providing a user interface on the glasses 100.

[0075] Figure 8This is a flowchart 800 illustrating a portion of a method for establishing a peer-to-peer Bluetooth communication link and aligning coordinate systems, according to some examples. For illustrative purposes, the operations of flowchart 800 are described herein as occurring serially or linearly. However, multiple operations of flowchart 800 may occur in parallel. Furthermore, the operations of flowchart 800 do not need to be executed in the order shown, and / or one or more boxes of flowchart 800 need not be executed and / or may be replaced by other operations.

[0076] The method begins at operation 802, where glasses 100 receives user input initiating a Bluetooth communication link with another device. In response, at operation 804, glasses 100 begins scanning for video feeds from a camera device within glasses 100.

[0077] If no QR code is detected in operation 806, the method returns to operation 804, and the glasses 100 continues scanning for QR codes. If a QR code is detected in operation 806, the glasses 100 captures an image of the QR code in box 808, decodes the QR code in operation 810, and establishes a Bluetooth communication link, as shown in flowchart 600.

[0078] In operation 812, if the QR code includes information specifying the pose of the first mobile device 402, the method proceeds to block 820, where a transformation between the captured image of the QR code and an undistorted version of the QR code is determined. The method then proceeds to operation 822.

[0079] In operation 810, if the QR code does not include information specifying the pose of the first mobile device 402, then in operation 814, the pose information is sent from the first mobile device 402 and received by the glasses 100. In operation 816, the glasses 100 captures an image of the QR code, and then in operation 818, a transformation between the captured image of the QR code and an undistorted version of the QR code is determined. The method then proceeds to operation 822, where the glasses 100 aligns the coordinate system as described above.

[0080] The glasses 100 continues to receive gesture updates and user input transmissions from the first mobile device 402 via a Bluetooth communication link. In operation 824, the gesture updates and user inputs are used by the glasses 100 as inputs from a controller with up to six degrees of freedom, as mentioned above.

[0081] As mentioned above, in flowcharts 500 and 800, the first mobile device 402, which shows the QR, typically plays a central role in the Bluetooth session, while the glasses 100 plays a peripheral role, although this is not necessary.

[0082] Depending on the application, an additional mobile device can scan a QR code 406 displayed by the first mobile device 402, which acts as the central device, to establish a communication link to the first mobile device 402 as a peripheral device. The gesture updates and user input received on the first mobile device 402 can then be provided to multiple glasses 100 or mobile devices.

[0083] Figure 9 This is a block diagram illustrating an example messaging system 900 for exchanging data (e.g., messages and associated content) over a network. The messaging system 900 includes multiple instances of user equipment 328 that host several applications, including a messaging client 902 and other applications 904. The messaging client 902 is communicatively coupled to other instances of the messaging client 902 (e.g., hosted on corresponding other user equipment 328), a messaging server system 906, and a third-party server 908 via a network 330 (e.g., the Internet). The messaging client 902 can also communicate with the locally hosted applications 904 using an application programming interface (API).

[0084] The messaging client 902 can communicate and exchange data with other messaging clients 902 and with the messaging server system 906 via the network 330. The data exchanged between messaging clients 902 and between messaging clients 902 and the messaging server system 906 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0085] The message transceiver server system 906 provides server-side functionality to a specific message transceiver client 902 via network 330. While some functions of the message transceiver system 900 are described herein as being performed by either the message transceiver client 902 or the message transceiver server system 906, the location of some functions within the message transceiver client 902 or the message transceiver server system 906 may be a design choice. For example, it may be technically preferred that some technologies and functions are initially deployed within the message transceiver server system 906, but that technology and functions are subsequently migrated to the message transceiver client 902 of user equipment 328 with sufficient processing capabilities.

[0086] The messaging server system 906 supports various services and operations provided to the messaging client 902. Such operations include transmitting data to and receiving data from the messaging client 902, and processing data generated by the messaging client 902. As an example, this data may include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and on-site event information. Data exchange within the messaging system 900 is activated and controlled through functions available via the user interface (UI) of the messaging client 902.

[0087] Specifically, the process now turns to message transceiver server system 906. Application Programming Interface (API) server 910 is coupled to application server 914 and provides a programming interface to application server 914. Application server 914 is communicatively coupled to database server 916, which facilitates access to database 920, which stores data associated with messages processed by application server 914. Similarly, web server 924 is coupled to application server 914 and provides a web-based interface to application server 914. For this purpose, web server 924 handles incoming network requests via Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0088] Application Programming Interface (API) server 910 receives and transmits message data (e.g., commands and message payloads) between user equipment 328 and application server 914. Specifically, API server 910 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by message transceiver client 902 to activate the functionality of application server 914. Application Programming Interface (API) server 910 exposes various functions supported by application server 914, including: account registration; login functionality; sending messages from a specific messaging client 902 to another messaging client 902 via application server 914, sending media files (e.g., images or videos) from messaging client 902 to messaging server 912, and providing possible access for another messaging client 902; setting up collections of media data (e.g., stories); retrieving the friend list of the user on user device 328; retrieving such collections; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends in the social graph; and opening application events (e.g., related to messaging client 902).

[0089] Application server 914 hosts several server applications and subsystems, including, for example, message transceiver server 912, image processing server 918, and social networking server 922. Message transceiver server 912 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of message transceiver client 902. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to message transceiver client 902. Given the hardware requirements for processing, additional processor- and memory-intensive data processing can also be performed on the server side by message transceiver server 912.

[0090] Application server 914 also includes image processing server 918, which is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at message transceiver server 912.

[0091] Social networking server 922 supports various social networking functions and services, and makes these functions and services available to messaging server 912. To this end, social networking server 922 maintains and accesses an entity graph within database 920. Examples of functions and services supported by social networking server 922 include identifying other users in messaging system 900 with whom a particular user has a relationship or who is "following," and also identifying a particular user's interests and other entities.

[0092] The messaging client 902 can notify the user of user device 328 or other users (e.g., "friends") associated with such a user of activities occurring in a shared or shareable session. For example, the messaging client 902 can provide participants in a conversation (e.g., a chat session) within the messaging client 902 with notifications related to the current or recent use of a game by one or more members of a user group. One or more users can be invited to join an active session or a new session can be initiated. In some examples, shared sessions can provide a shared augmented reality experience that multiple people can collaborate on or participate in.

[0093] Figure 10This is a block diagram 1000 illustrating a software architecture 1004 that can be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware, such as a machine 1002 including a processor 1020, memory 1026, and I / O components 1038. In this example, the software architecture 1004 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1004 includes layers such as an operating system 1012, libraries 1008, a framework 1010, and an application 1006. Operationally, the application 1006 invokes API call 1050 through the software stack and receives a message 1052 in response to API call 1050.

[0094] Operating system 1012 manages hardware resources and provides public services. Operating system 1012 includes, for example, a core 1014, services 1016, and drivers 1022. Core 1014 serves as an abstraction layer between hardware and other software layers. For example, core 1014 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1016 can provide other public services to other software layers. Driver 1022 is responsible for controlling or interfacing with the underlying hardware. For example, driver 1022 may include a display driver, a camera driver, a BLUETOOTH® or BLUETOOTH® low-power driver, a flash memory driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), a Wi-Fi® driver, an audio driver, a power management driver, etc.

[0095] Library 1008 provides low-level common infrastructure used by application 1006. Library 1008 may include system library 1018 (e.g., C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1008 may include API libraries 1024, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., OpenGL framework for rendering graphical content in two-dimensional (2D) and three-dimensional (3D) on a display, GLMotif for implementing a 3D user interface), image feature extraction libraries (e.g., OpenIMAJ), database libraries (e.g., SQLite providing various relational database functionalities), web libraries (e.g., WebKit providing web browsing functionality), etc. Library 1008 may also include various other libraries 1028 to provide many other APIs to application 1006.

[0096] Framework 1010 provides high-level common infrastructure for use by application 1006. For example, Framework 1010 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 1010 can provide a wide range of other APIs that can be used by application 1006, some of which may be specific to a particular operating system or platform.

[0097] In the example, application 1006 may include home application 1036, contacts application 1030, browser application 1032, book reader application 1034, location application 1042, media application 1044, messaging application 1046, game application 1048, and a variety of other applications such as third-party application 1040. Application 1006 is a program that performs the functions defined in the program. One or more applications 1006 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1040 (e.g., an application developed by an entity other than a platform vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software that runs on a mobile operating system such as iOS™, Android™, Windows® Phone, or another mobile operating system. In this example, a third-party application 1040 can invoke API call 1050 provided by the operating system 1012 to facilitate the functionality described herein.

[0098] Figure 11This is a schematic representation of machine 1100 or computing device in which instructions 1110 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1100 to perform any or more of the methods discussed herein. For example, instructions 1110 can cause machine 1100 to perform any or more of the methods described herein. Instructions 1110 transform a general, unprogrammed machine 1100 into a specific machine 1100 programmed to perform the described and illustrated functions in the described manner. Machine 1100 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1100 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1100 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, head-mounted devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1110 specifying actions to be taken by machine 1100. Furthermore, while a single machine 1100 is shown, the term "machine" may also be considered to include a collection of machines that individually or jointly execute instructions 1110 to perform any one or more of the methods discussed herein.

[0099] Machine 1100 may include processor 1102, memory 1104, and I / O unit 1106, which may be configured to communicate with each other via bus 1144. In the example, processor 1102 (e.g., a central processing unit (CPU), a simplified instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1108 and processor 1112 that execute instruction 1110. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although... Figure 11 Multiple processors 1102 are shown, but machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0100] Memory 1104 includes main memory 1114, static memory 1116, and storage cells 1118, all of which are accessible by processor 1102 via bus 1144. Main memory 1104, static memory 1116, and storage cells 1118 store instructions 1110 that implement any one or more of the methods or functions described herein. During execution of instructions 1110 by networked system 300, instructions 1110 may also reside wholly or partially in main memory 1114, in static memory 1116, in machine-readable medium 1120 within storage cell 1118, in one or more processors of processor 1102 (e.g., in the processor's cache memory), or in any suitable combination thereof.

[0101] I / O component 1106 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 1106 included in a particular machine will depend on the machine type. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 1106 may include... Figure 11 Many other components are not shown. In various examples, I / O component 1106 may include output component 1128 and input component 1132. Output component 1128 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input component 1132 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing positioning and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0102] In another example, I / O component 1106 may include: biometric component 1134, motion component 1136, environmental component 1138, or position component 1140, and various other components. For example, biometric component 1134 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and recognizing people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1136 includes accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1138 includes, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases or measuring pollutants in the atmosphere for safety purposes), or other components that can provide indications, measurement results, or signals corresponding to the surrounding physical environment. Position component 1140 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure to obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0103] A wide variety of technologies can be used to implement communication. I / O component 1106 also includes communication component 1142, which is operable to couple the networking system 300 to network 1122 or device 1124 via coupling 1130 and coupling 1126, respectively. For example, communication component 1142 may include a network interface component or another suitable device that interfaces with network 1122. In further examples, communication component 1142 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components that provide communication via other modalities. Device 1124 may be another machine or any peripheral device from a wide variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0104] Furthermore, the communication component 1142 may detect identifiers or include components operable to detect identifiers. For example, the communication component 1142 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting optical sensors such as one-dimensional barcodes, such as Universal Product Code (UPC) barcodes; multi-dimensional barcodes, such as Quick Response (QR) codes, Aztec codes, data matrices, dataglyphs, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be obtained via the communication component 1142, such as location obtained via Internet Protocol (IP) geolocation, location obtained via Wi-Fi® signal triangulation, location obtained by detecting NFC beacon signals that can indicate a specific location, etc.

[0105] Various memories (e.g., memory 1104, main memory 1114, static memory 1116, and / or the memory of processor 1102) and / or storage units 1118 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1110) cause various operations to implement the disclosed examples when executed by processor 1102.

[0106] Instructions 1110 can be transmitted or received over network 1122 via a network interface device (e.g., a network interface component included in communication component 1142), using a transmission medium and any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1110 can be transmitted or received to device 1124 via a transmission medium through coupling 1126 (e.g., peer-to-peer coupling).

[0107] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be transmitted or received over a network using a transmission medium via a network interface device.

[0108] "User equipment" or "client equipment" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other user equipment or client devices. User equipment or client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0109] "Communications network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution (EDGE) for GSM, 3rd Generation Partnership Project (3GPP) including 3G, 4th Generation Wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0110] A “component” means a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed to be used as part of a program that performs a specific function typically related to it. A component can constitute a software component (e.g., code implemented on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing some operations and can be configured or arranged in a particular physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform some operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform some operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) tailored to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to implement a hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., software-configured) circuit system can be driven by cost and time considerations. Accordingly, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform some of the operations described herein. Considering the example of a hardware component being temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, this general-purpose processor can be configured at different times as its respective dedicated processor (e.g., including different hardware components). The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission (e.g., via appropriate circuitry and buses) between or within two or more hardware components. In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). The various operations of the example methods described herein can be performed by one or more processors configured temporarily (e.g., via software) or permanently to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be partially implemented by processors, where one or more specific processors are examples of hardware. For example, some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, some operations can be performed by a group of computers (as an example of a machine including processors), where these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, reside within a single machine, or be deployed across multiple machines. In some examples, the processors or processor-implemented components can reside in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other examples, the processors or processor-implemented components can be distributed across multiple geographic locations.

[0111] "Computer-readable medium" or "machine storage medium" refers to one or more non-transitory storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. This term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are covered by the term "signal medium."

[0112] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals that are applied to operate a machine. For example, a processor can be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes called "cores") capable of executing instructions simultaneously.

[0113] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" can be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0114] Furthermore, according to embodiments of this disclosure, the following configurations 1-56 are provided.

[0115] 1. A computer-implemented method, executed by a first mobile device, for establishing a communication link between the first mobile device and a second mobile device, the method comprising:

[0116] An image of machine-readable code captured by the first mobile device and displayed on the second mobile device;

[0117] Image decoding communication settings information captured by the first mobile device from the machine-readable code; and

[0118] The communication settings information is used to establish a communication link with the second mobile device.

[0119] 2. The computer-implemented method according to configuration 1, wherein the communication settings information is dynamically generated.

[0120] 3. The computer-implemented method according to configuration 1 further includes:

[0121] Send a message broadcasting at least a portion of the communication settings information.

[0122] 4. The computer-implemented method according to configuration 1 further includes:

[0123] The system scans for messages from the second mobile device, which broadcast at least a portion of the communication settings information.

[0124] 5. The computer-implemented method according to configuration 1 further includes:

[0125] Determine the transformation of the reference markers displayed by the second mobile device;

[0126] Receive information specifying the posture of the second mobile device;

[0127] The relative pose between the first mobile device and the second mobile device is determined using the transformation and the information specifying the pose of the second mobile device; and

[0128] Align the coordinate system between the first mobile device and the second mobile device.

[0129] 6. The computer-implemented method according to configuration 5, wherein the machine-readable code is the reference mark.

[0130] 7. The computer-implemented method according to configuration 6, wherein the information specifying the posture of the second mobile device is encoded in the machine-readable code.

[0131] 8. The computer-implemented method according to configuration 6 further includes:

[0132] Receive updates to the information regarding the posture of the designated second mobile device via the communication link; and

[0133] User interface actions are performed on the first mobile device based on the received update of the information regarding the gesture of the designated second mobile device.

[0134] 9. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions, which, when executed by a first mobile device, cause the first mobile device to perform operations for establishing a communication link between the first mobile device and a second mobile device, the operations comprising:

[0135] An image of machine-readable code captured by the first mobile device and displayed on the second mobile device;

[0136] Image decoding communication settings information captured by the first mobile device from the machine-readable code; and

[0137] The communication settings information is used to establish a communication link with the second mobile device.

[0138] 10. The non-transitory computer-readable storage medium according to configuration 9, wherein the communication settings information is dynamically generated.

[0139] 11. The non-transitory computer-readable storage medium according to configuration 9, wherein the operation further includes;

[0140] Send a message broadcasting at least a portion of the communication settings information.

[0141] 12. The non-transitory computer-readable storage medium according to configuration 9, wherein the operation further includes:

[0142] Determine the transformation of the reference markers displayed by the second mobile device;

[0143] Receive information specifying the posture of the second mobile device;

[0144] The relative pose between the first mobile device and the second mobile device is determined using the transformation and the information specifying the pose of the second mobile device; and

[0145] Align the coordinate system between the first mobile device and the second mobile device.

[0146] 13. The non-transitory computer-readable storage medium according to configuration 12, wherein the machine-readable code is the reference mark.

[0147] 14. The non-transitory computer-readable storage medium according to configuration 12, wherein the operation further includes:

[0148] Receive updates to the information regarding the posture of the designated second mobile device via the communication link; and

[0149] User interface actions are performed on the first mobile device based on the received update of the information regarding the gesture of the designated second mobile device.

[0150] 15. A computing device, comprising:

[0151] Processor; and

[0152] A memory storing instructions that, when executed by the processor, configure the device to perform operations for establishing a communication link between a first mobile device and a second mobile device, the operations including:

[0153] An image of machine-readable code captured by the first mobile device and displayed on the second mobile device;

[0154] Image decoding communication settings information captured by the first mobile device from the machine-readable code; and

[0155] The communication settings information is used to establish a communication link with the second mobile device.

[0156] 16. The computing device according to configuration 15, wherein the communication setting information is dynamically generated.

[0157] 17. The computing device according to configuration 15, wherein the operation further includes:

[0158] Scan for at least a portion of the messages from the second mobile device that broadcast the communication settings information.

[0159] 18. The computing device according to configuration 15, wherein the operation further includes:

[0160] Determine the transformation of the reference markers displayed by the second mobile device;

[0161] Receive information specifying the posture of the second mobile device;

[0162] The relative pose between the first mobile device and the second mobile device is determined using the transformation and the information specifying the pose of the second mobile device; and

[0163] Align the coordinate system between the first mobile device and the second mobile device.

[0164] 19. The computing device according to configuration 18, wherein the machine-readable code is the reference mark.

[0165] 20. The computing device according to configuration 19, wherein the operation further includes:

[0166] Receive updates to the information regarding the posture of the designated second mobile device via the communication link; and perform user interface actions on the first mobile device based on the received updates to the information regarding the posture of the designated second mobile device.

[0167] Changes and modifications may be made to the disclosed examples without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure as set forth in the appended claims.

Claims

1. A computer-implemented method comprising: capturing, by a first mobile device, an image displayed on a second mobile device that includes a fiducial marker; determining a relative pose between the first mobile device and the second mobile device by generating a pose transform of the fiducial marker displayed by the second mobile device; receiving information specifying an update to a pose of the second mobile device; and processing the received information specifying an update to a pose of the second mobile device as user input to the first mobile device.

2. The computer-implemented method of claim 1, further comprising: receiving, by the first mobile device, touch input received on the second mobile device; and processing the received touch input as additional user input to the first mobile device.

3. The computer-implemented method of claim 2, wherein: the touch input is processed to select an augmented reality object displayed by the first mobile device, and the update to a pose of the second mobile device is processed to move the augmented reality object displayed by the first mobile device. the touch input is processed to move a cursor displayed by the first mobile device. the information specifying an update to a pose of the second mobile device is received over a communication link.

4. The computer-implemented method of claim 2, wherein, the image includes communication session setup information.

5. The computer-implemented method of claim 1, wherein, 7. The computer-implemented method of claim 6, further comprising:

6. The computer-implemented method of claim 1, wherein, receiving a message that broadcasts at least a portion of the communication session setup information.

8. The computer-implemented method of claim 6, further comprising: scanning for a message from the second mobile device that broadcasts at least a portion of the communication session setup information.

9. The computer-implemented method of claim 6, further comprising: scanning, by a third mobile device, for the communication session setup information; and joining, by the third mobile device, a communication session that includes the first mobile device and the second mobile device. the communication session includes a Bluetooth mesh topology.

11. A computing apparatus comprising: a processor; and 10. The computer-implemented method of claim 9, wherein, memory storing instructions that, when executed by the processor, configure the apparatus to perform operations comprising: capturing, by a first mobile device, an image displayed on a second mobile device that includes a fiducial marker; determining a relative pose between the first mobile device and the second mobile device by generating a pose transform of the fiducial marker displayed by the second mobile device; receiving information specifying an update to a pose of the second mobile device; and processing the received information specifying an update to a pose of the second mobile device as user input to the first mobile device. the operations further comprising: receiving, by the first mobile device, touch input received on the second mobile device; and processing the received touch input as additional user input to the first mobile device.

13. The computing apparatus of claim 12, wherein:

12. The computing device of claim 11, wherein, the touch input is processed to select an augmented reality object displayed by the first mobile device, and the update to a pose of the second mobile device is processed to move the augmented reality object displayed by the first mobile device. ​ ​ ​ The update to the pose of the second mobile device is processed to move the augmented reality object displayed by the first mobile device.

14. The computing device of claim 12, wherein, The touch input is processed to move a cursor displayed by the first mobile device.

15. The computing device of claim 11, wherein, The information specifying the update to the pose of the second mobile device is received over a communication link.

16. The computing device of claim 11, wherein, The image includes communication session setup information.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed by a first mobile device, cause the first mobile device to perform operations comprising: capturing, by the first mobile device, an image displayed on a second mobile device that includes a fiducial marker; determining a relative pose between the first mobile device and the second mobile device by generating a pose transform of the fiducial marker displayed by the second mobile device; receiving information specifying an update to a pose of the second mobile device; and processing the received information specifying the update to the pose of the second mobile device as user input to the first mobile device.

18. The non-transitory computer-readable storage medium of claim 17, wherein, The operations further comprise: receiving, by the first mobile device, touch input received on the second mobile device; and processing the received touch input as additional user input to the first mobile device.

19. The non-transitory computer-readable storage medium of claim 18, wherein: the touch input is processed to select an augmented reality object displayed by the first mobile device, and the update to the pose of the second mobile device is processed to move the augmented reality object displayed by the first mobile device.

20. The non-transitory computer-readable storage medium of claim 18, wherein, The touch input is processed to move a cursor displayed by the first mobile device.

Images