Secure peer-to-peer connection between mobile devices

Abstract

A communication link is established between the first mobile device and the second mobile device using communication setting information in a machine-readable code displayed on a display of the second mobile device. The first mobile device captures an image of the machine-readable code and decodes the image of the machine-readable code to extract dynamically generated communication setting information. A communication link is then established between the two devices using the communication setup information. The machine-readable code may also be used as a fiducial marker to establish an initial relative pose between the two devices. The gesture update then received from the second mobile device may be used as a user interface input to the first mobile device.

Description

[0001] Related application data

[0002] This application claims priority to U.S. Patent Application No. 17 / 813,901, filed on July 20, 2022, the contents of which are incorporated herein by reference as if expressly set forth. Technical Field

[0003] The present disclosure relates generally to communications between mobile devices and, more particularly, to establishing secure peer-to-peer connections between mobile devices. Background Art

[0004] For non-technical people, establishing a secure communication link between nearby mobile devices usually involves an arduous setup process. Some solutions, such as Apple's AirDrop, use Bluetooth broadcasts to identify mobile devices in the area. If a user wants to share a file, a list of potential recipients in the area is displayed.

[0005] The use of Bluetooth broadcasts and the associated information exchange is a possible security risk and also requires the user to specifically enable or disable the file sharing feature to allow all devices in the area, disallow devices in the area, or allow compatible devices of contacts in the area for initial connection and sharing requests. In addition, proprietary implementations such as AirDrop have limited use outside of their ecosystem.

[0006] In addition, when pairing devices based on Bluetooth broadcasts, a list of available devices is initially presented to the user for manual selection. Especially in the case where there are more than one device of the same brand and model, knowing which device to select from the list can be challenging. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0008] Figure 1 is a stereoscopic diagram of a head mounted device according to some examples.

[0009] Figure 2 According to some examples Figure 1 Another view of the headset.

[0010] Figure 3 It is shown that according to some examples, Figure 1 A block diagram of the networking system with details of the head mounted device.

[0011] Figure 4 is a basic process flow diagram illustrating establishment of a peer-to-peer Bluetooth communication link according to some examples.

[0012] Figure 5 is a flow chart illustrating a portion of a method of establishing a peer to peer Bluetooth communications link according to some examples.

[0013] Figure 6 is a flow chart illustrating a portion of a method of establishing a peer to peer Bluetooth communications link according to some examples.

[0014] Fig. 7A is a view of an environment through a user's glasses, wherein a first mobile device is displaying a QR code, according to some examples.

[0015] Figure 7B is a view of an environment through glasses of a user, wherein a first mobile device is being used as an input device for the glasses, according to some examples.

[0016] Figure 8 is a flow chart illustrating a portion of a method of establishing a peer-to-peer Bluetooth communication link and aligning a coordinate system according to some examples.

[0017] Fig. 9 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0018] Fig.10 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to some examples.

[0019] Fig.11 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0020] In some examples, the establishment of a secure peer-to-peer connection between mobile devices is achieved by displaying a QR code on a display of a first mobile device. The QR code is scanned by a second mobile device, and then the second mobile device establishes a connection with the first mobile device based on the information encoded in the QR code. In some cases, the first mobile device is a smart phone, and the second mobile device is a head-mounted augmented reality (AR) device.

[0021] A QR code is a machine-readable optical label that can contain information about the item to which it is attached. QR codes are widely used to share small amounts of data between mobile devices. For example, some messaging applications use QR codes to quickly allow a user displaying the QR code to be quickly added as a "friend" of the user scanning the mobile device. QR codes are also often encoded with a URL, which allows the mobile device to retrieve additional information from a remote server.

[0022] As disclosed herein, the QR code displayed on the second device includes Bluetooth Low Energy (BLE) address information, which can be used to establish a BLE communication link between two devices in an ad hoc connection without any other setup. In the establishment of a traditional Bluetooth connection, the user of the mobile device initially selects the device from a list of available devices and thus indicates it as trusted. Then, in a process called bonding, the static address and other information of the selected Bluetooth device are stored in the memory of the mobile device. Binding is the exchange of long-term keys after pairing occurs and the storage of these keys for later use. Binding creates a persistent security relationship between two devices. Binding helps to re-establish a connection between trusted devices because the trusted devices have been so designated.

[0023] Using the scan of a QR code displayed on one device to pair two devices avoids the requirement that binding occurs or has occurred, since the user will presumably only scan the QR code on the mobile device with which it intends to connect (i.e., a device trusted by the user). The Bluetooth address used can be dynamic, since there is no need to rely on the address of the paired device being previously stored in memory and therefore required to be static to allow connection. In some cases, a "confirm connection" user interface prompt may be provided on both devices to prevent accidental scanning, which is unlikely in any case.

[0024] In some examples, a computer-implemented method performed by a first mobile device for establishing a communication link between the first mobile device and a second mobile device is provided, the method comprising: capturing, by the first mobile device, an image of a machine-readable code displayed on the second mobile device; decoding, by the first mobile device, communication setup information from the captured image of the machine-readable code; and establishing a communication link with the second mobile device using the communication setup information. The communication setup information may be dynamically generated.

[0025] The computer-implemented method may also include sending a message broadcasting at least a portion of the communication settings information and / or scanning for a message broadcasting at least a portion of the communication settings information from a second mobile device.

[0026] In some examples, the method may include: determining a transformation of a fiducial marker displayed by a second mobile device; receiving information specifying a posture of the second mobile device; determining a relative posture between the first mobile device and the second mobile device using the transformation and the information specifying the posture of the second mobile device; and aligning a coordinate system between the first mobile device and the second mobile device. The machine-readable code may be the fiducial marker. The information specifying the posture of the second mobile device may be encoded in the machine-readable code.

[0027] The method may also include receiving an update to the information specifying the gesture of the second mobile device via the communication link; and performing a user interface action on the first mobile device based on the received update to the information specifying the gesture of the second mobile device.

[0028] In some examples, a non-transitory computer-readable storage medium is provided, which includes instructions that, when executed by a first mobile device, cause the first mobile device to perform operations for establishing a communication link between the first mobile device and a second mobile device according to any of the methods and limitations described above, the operations including but not limited to: capturing, by the first mobile device, an image of a machine-readable code displayed on the second mobile device; decoding, by the first mobile device, communication setup information from the captured image of the machine-readable code; and establishing a communication link with the second mobile device using the communication setup information.

[0029] In some examples, a computing device is provided that includes a processor and a memory storing instructions that, when executed by the processor, configure the device to perform operations for establishing a communication link between a first mobile device and a second mobile device according to any of the methods and limitations described above, the operations including but not limited to: capturing, by the first mobile device, an image of a machine-readable code displayed on the second mobile device; decoding, by the first mobile device, communication setup information from the captured image of the machine-readable code; and establishing a communication link with the second mobile device using the communication setup information.

[0030] Other technical features may be readily apparent to those skilled in the art from the accompanying drawings, descriptions and claims.

[0031] Figure 1 1 is a stereogram of a head-mounted AR device (e.g., glasses 100) according to some examples. Glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0032] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or integrated construction.

[0033] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 302 discussed below.

[0034] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0035] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, positioning sensors, motion sensors, etc.

[0036] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from a real-world scene.

[0037] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is typically arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, typically vertically aligned means that the touchpad is more vertical than horizontal, although potentially more vertical than this. Additional user input may be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from a user of the glasses 100.

[0038] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0039] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0040] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 superimposed with the view of the real world seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 superimposed with the view of the real world seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's real-world view, including displaying a 3D user interface to the user of the glasses 100.

[0041] However, it should be understood that other display technologies or configurations may be used within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0042] In use, the user of the glasses 100 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 3 The user device 328 shown in FIG. 3 may be used to interact with the glasses 100 through voice input or touch input on the glasses 100 and / or hand movements, positioning, and location detected by the glasses 100.

[0043] Figure 3 is a block diagram showing a networked system 300 including details of the glasses 100 according to some examples. The networked system 300 includes the glasses 100, a user device 328, and a server system 332. The user device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low power wireless connection 336 and / or a high speed wireless connection 334. The user device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The server system 332 may be a server system 332 that is a networked computing system that may be used separately. Fig.10 and Fig.11 The details of the software architecture 1004 or machine 1100 described in the implementation of the user device 328 and any elements of the server system 332 and network 330.

[0044] The glasses 100 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include microphones, audio speakers, biometric sensors, additional sensors, or additional display elements integrated with the data processor 302. Fig.10 and Fig.11 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O components 1106, including output components 1128, motion components 1136, etc. Figure 2 Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the user's left eye and right eye.

[0045] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0046] Interface 312 refers to any source of user commands provided to data processor 302. In one or more examples, interface 312 is a physical button that, when pressed, sends a user input signal from interface 312 to low-power processor 314. Low-power processor 314 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. Low-power processor 314 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing a button for a long period of time can capture a still image. In some examples, interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera 308. In other examples, interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from user device 328.

[0047] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to user device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0048] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from the user device 328 via the low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart (also known as Bluetooth TMLow power wireless communication (LPW) is one standard implementation of a low power wireless communication system that may be used to implement low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0049] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 302. High-speed processor 322 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.10 The high-speed processor 322, which executes the software architecture of the data processor 302, manages data transmission with the high-speed wireless circuit system 326, in addition to any other duties. In some examples, the high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0050] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, stand-alone element of data processor 302. In some such examples, electrical wiring may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will direct high-speed processor 322 whenever a read or write operation involving memory 324 is desired.

[0051] The tracking module 340 estimates the position and orientation ("pose") of the glasses 100. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the position component 1140, as well as GPS data, to track the location and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100 that indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0052] The GPU and display driver 338 may use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100 that reflects changes in the user's position and orientation with respect to physical objects in the user's real-world environment.

[0053] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the user device 328 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1006, such as the messaging application 1046.

[0054] Figure 4 4 is a basic process flow diagram illustrating the establishment of a peer-to-peer Bluetooth communication link according to some examples. As can be seen in the figure, a QR code 406 is displayed by a first mobile device 402. The QR code 406 has Bluetooth address information 408 encoded therein, which has been dynamically generated. The QR code is scanned and decoded by a second mobile device 404. Using the address information 408, the second mobile device 404 initiates a Bluetooth communication link 410 with the first mobile device 402. The Bluetooth communication link 410 enables the exchange of data 412 between the two devices. Figure 5 The first mobile device 402 displaying the QR code typically assumes a central role in the Bluetooth session, while the second mobile device 404 assumes a peripheral role, although this is not required.

[0055] The address information 408 includes a dynamically generated BLE service / characteristic UUID (Universally Unique Identifier) ​​for establishing a communication link. Therefore, the address information 408 is unique and is clearly defined in each application running on each mobile device. Conventionally, if two or more people in the same area are wearing glasses 100 and are trying to connect their glasses 100 to their user devices 328, a conflict will occur. The current method avoids this problem, allowing all users in the same area to have a unique and secure communication link between their own devices.

[0056] Figure 55 is a flowchart 500 illustrating a portion of a method for establishing a peer-to-peer Bluetooth communication link performed on a first mobile device 402 according to some examples. For illustrative purposes, the operations of the flowchart 500 are described herein as occurring serially or linearly. However, multiple operations of the flowchart 500 may occur in parallel. In addition, the operations of the flowchart 500 do not need to be performed in the order shown and / or one or more blocks of the flowchart 500 do not need to be performed and / or may be replaced by other operations.

[0057] The method begins at operation 502, where the first mobile device 402 receives user input to initiate a Bluetooth communication link with another device. In response, in operation 504, the first mobile device 402 generates a random UUID as a BLE service / characteristic ID. In operation 506, the first mobile device 402 generates a QR code embodying the UUID and displays the QR code in operation 508. The first mobile device 402 then begins scanning for the UUID to be broadcast in a message sent by the second mobile device 404 using its Bluetooth transceiver.

[0058] If no UUID is detected in operation 512, the method returns to operation 510, and the first mobile device 402 continues to scan for messages broadcasting at least some of the UUIDs sent by the second mobile device 404. If a UUID is detected in operation 512, the first mobile device 402 responds to the second mobile device 404 and establishes a secure Bluetooth communication link in operation 514. Then, in operation 516, the first mobile device 402 exchanges data with the second mobile device 404.

[0059] Figure 6 600 is a flowchart illustrating a portion of a method for establishing a peer-to-peer Bluetooth communication link according to some examples. For illustrative purposes, the operations of flowchart 600 are described herein as occurring serially or linearly. However, multiple operations of flowchart 600 may occur in parallel. In addition, the operations of flowchart 600 do not need to be performed in the order shown and / or one or more blocks of flowchart 600 do not need to be performed and / or may be replaced by other operations.

[0060] The method begins at operation 602 , where the second mobile device 404 receives a user input to initiate a Bluetooth communication link with another device. In response, in operation 604 , the second mobile device 404 begins scanning for a video feed from a camera in the second mobile device 404 .

[0061] If the QR code is not detected in operation 606, the method returns to operation 604, and the second mobile device 404 continues to scan for the QR code. If the QR code is detected in operation 606, the second mobile device 404 decodes the QR code in operation 608 to extract the UUID reflected therein. In operation 610, the second mobile device 404 uses its Bluetooth transceiver to send a message broadcasting at least some of the decoded UUIDs. In operation 612, the second mobile device 404 receives a confirmation transmission from the first mobile device 402 in response to the message, establishes a Bluetooth communication link in operation 614, and begins exchanging data in operation 616.

[0062] It should be understood that instead of sending the UUID in operations 610 and 612, the second mobile device 404 may perform the following operations in flowchart 600: Figure 5 Instead of scanning for the UUID in operations 510 and 512, the first mobile device 402 in flowchart 500 may perform the following operations: Figure 6 Operations 610 and 612 in FIG. 5 send the UUID.

[0063] As described above, in flowchart 500 and flowchart 600, first mobile device 402 displaying a QR typically assumes a central role in a Bluetooth session, while second mobile device 404 assumes a peripheral role, although this is not required.

[0064] Depending on the application, additional mobile devices may scan the QR code 406 displayed by the first mobile device 402 acting in the central role to establish a communication link as a peripheral device to the first mobile device 402. Alternatively, multiple mobile devices may be connected together in a BLE mesh topology.

[0065] Fig. 7A 702 through the user's glasses 100, where the first mobile device 402 is displaying a QR code 704. In this case, in addition to the first mobile device 402 and the glasses 100 (as the second mobile device) executing Figure 5 and Figure 6 In addition to the method, the QR code 704 is also used to align the coordinate systems of the two devices.

[0066] Both the first mobile device 402 and the glasses 100 perform visual inertial odometry to determine and track their respective positions and orientations (“poses”) within the environment 702. The relative pose between the glasses 100 and the first mobile device 402 can be determined by having the glasses 100 capture an image of a fiducial marker. In this case, the QR code serves as a fiducial marker in addition to providing information for establishing a communication link. Since the shape and correct orientation of the fiducial marker are known, the glasses 100 can determine the pose transformation between the captured image of the fiducial marker and the actual fiducial marker.

[0067] In some examples, the pose of the first mobile device 402 can be encoded in the QR code 704, so that the glasses 100 can capture an image of the QR code 704, determine a transformation of the image of the QR code 704, and determine the relative pose between the first mobile device 402 and the glasses 100. In other examples, the pose of the first mobile device 402 is reported to the glasses 100 via the Bluetooth communication link 410 after the Bluetooth communication link 410 has been established, while continuing to display the QR code. The glasses 100 can then capture the image of the QR code and the current pose of the first mobile device 402 at almost the same time. The latter method is less sensitive to the movement of the first mobile device 402, which may occur after the QR code including the pose of the mobile device has been generated but before it has been captured by the glasses 100.

[0068] The determined transformation will provide the relative pose between the first mobile device 402 and the glasses 100. Either or both of the coordinate systems of the first mobile device 402 and the glasses 100 may then be adjusted so that the poses of the two devices are expressed in the same coordinate system. The first mobile device 402 then reports its poses and touch inputs to the glasses 100 via the Bluetooth communication link 410.

[0069] Fig. 7A Also shown is a text prompt 706 displayed by the glasses 100 to the user, which instructs the user to hold the phone in front of the glasses 100 so that the QR code 406 can be imaged and decoded.

[0070] Figure 7B is a view of an environment 702 through a user's glasses 100, wherein a first mobile device 402 is being used as an input device for the glasses 100, according to some examples. Figure 7B In the embodiment of the present invention, a Bluetooth communication link has been established and the coordinate systems of the two devices are aligned as described above. The first mobile device 402 is reporting its posture and any user touch input received to the glasses 100 via the Bluetooth communication link 410.

[0071] As shown, the first mobile device 402 is displaying user interface elements 708, including, for example, a directional touch pad that can be used to move a cursor 714 displayed by the glasses 100. A user interface window 710 being displayed by the glasses 100 is also shown.

[0072] The user can interact with augmented reality elements displayed by the glasses, such as AR elements 712 and window 710, using a combination of touch input and gestures using the first mobile device 402. For example, using a directional touch pad shown on the touch screen of the first mobile device 402, the user can place a cursor 714 over the window 710. The button shown in the window can then be activated with a single touch on the touch screen of the first mobile device 402, or the window can be moved within the field of view of the glasses 100 by pressing and holding on the touch screen and then pointing the first mobile device 402 to the desired location of the window 710.

[0073] Thus, the first mobile device 402 functions as a six degree of freedom controller with touch input for the glasses 100. This alleviates some of the challenges of providing a user interface on the glasses 100.

[0074] Figure 8 800 is a flowchart illustrating a portion of a method for establishing a peer-to-peer Bluetooth communication link and aligning a coordinate system according to some examples. For illustrative purposes, the operations of flowchart 800 are described herein as occurring serially or linearly. However, multiple operations of flowchart 800 may occur in parallel. In addition, the operations of flowchart 800 do not need to be performed in the order shown and / or one or more blocks of flowchart 800 do not need to be performed and / or may be replaced by other operations.

[0075] The method begins at operation 802 where the glasses 100 receive a user input to initiate a Bluetooth communication link with another device. In response, in operation 804 the glasses 100 begin scanning a video feed from a camera in the glasses 100.

[0076] If the QR code is not detected in operation 806, the method returns to operation 804 and the glasses 100 continue to scan for QR codes. If the QR code is detected in operation 806, the glasses 100 capture an image of the QR code in box 808, decode the QR code in operation 810 and establish a Bluetooth communication link, as in flowchart 600.

[0077] In operation 812 , if the QR code includes information specifying a pose of the first mobile device 402 , the method proceeds to block 820 where a transformation between the captured image of the QR code and an undistorted version of the QR code is determined. The method then proceeds to operation 822 .

[0078] In operation 810, if the QR code does not include information specifying the pose of the first mobile device 402, then in operation 814, the pose information is sent from the first mobile device 402 and received by the glasses 100. In operation 816, the glasses 100 capture an image of the QR code, and then determine a transformation between the captured image of the QR code and an undistorted version of the QR code in operation 818. The method then proceeds to operation 822, where the glasses 100 align the coordinate system as described above.

[0079] The glasses 100 continue to receive transmissions of gesture updates and user inputs from the first mobile device 402 via the Bluetooth communications link. In operation 824, the gesture updates and user inputs are used by the glasses 100 as inputs from the up to six degrees of freedom controller as mentioned above.

[0080] As also mentioned above, in flowchart 500 and flowchart 800, the first mobile device 402 displaying the QR typically assumes the central role in the Bluetooth session, while the glasses 100 assume the peripheral role, although this is not required.

[0081] Depending on the application, the additional mobile device can scan the QR code 406 displayed by the first mobile device 402 acting in the central role to establish a communication link as a peripheral device to the first mobile device 402. Gesture updates and user input received on the first mobile device 402 can then be provided to multiple glasses 100 or mobile devices.

[0082] Fig. 9 is a block diagram illustrating an example messaging system 900 for exchanging data (e.g., messages and associated content) over a network. The messaging system 900 includes multiple instances of user devices 328 that host several applications including a messaging client 902 and other applications 904. The messaging client 902 is communicatively coupled to other instances of the messaging client 902 (e.g., hosted on respective other user devices 328), a messaging server system 906, and a third-party server 908 via a network 330 (e.g., the Internet). The messaging client 902 may also communicate with the locally hosted application 904 using an application program interface (API).

[0083] The messaging clients 902 are able to communicate and exchange data with other messaging clients 902 and with a messaging server system 906 via the network 330. The data exchanged between the messaging clients 902 and between the messaging clients 902 and the messaging server system 906 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0084] The messaging server system 906 provides server-side functionality to a particular messaging client 902 via the network 330. Although some functions of the messaging system 900 are described herein as being performed by the messaging client 902 or by the messaging server system 906, it may be a design choice whether some functions are located within the messaging client 902 or within the messaging server system 906. For example, it may be technically preferred to initially deploy some technologies and functions within the messaging server system 906, but then migrate the technologies and functions to the messaging client 902 where the user device 328 has sufficient processing power.

[0085] The messaging server system 906 supports various services and operations provided to the messaging client 902. Such operations include transmitting data to the messaging client 902, receiving data from the messaging client 902, and processing data generated by the messaging client 902. As examples, the data may include message content, user device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 900 is activated and controlled by functions available via the user interface (UI) of the messaging client 902.

[0086] Turning now specifically to the messaging server system 906, an application program interface (API) server 910 is coupled to and provides a programming interface to an application server 914. The application server 914 is communicatively coupled to a database server 916, which facilitates access to a database 920 that stores data associated with messages processed by the application server 914. Similarly, a web server 924 is coupled to and provides a web-based interface to the application server 914. To this end, the web server 924 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0087] The application program interface (API) server 910 receives and transmits message data (e.g., commands and message payloads) between the user device 328 and the application server 914. Specifically, the application program interface (API) server 910 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 902 to activate the functions of the application server 914. The application program interface (API) server 910 exposes various functions supported by the application server 914, including: account registration; login functionality; sending messages from a particular messaging client 902 to another messaging client 902 via the application server 914, sending media files (e.g., images or videos) from the messaging client 902 to the messaging server 912, and possible access by another messaging client 902; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the user device 328; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 902).

[0088] The application server 914 hosts several server applications and subsystems, including, for example, a messaging server 912, an image processing server 918, and a social network server 922. The messaging server 912 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 902. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 902. In view of the hardware requirements for processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 912.

[0089] The application server 914 also includes an image processing server 918 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 912 .

[0090] The social network server 922 supports various social networking functions and services and makes these functions and services available to the messaging server 912. To this end, the social network server 922 maintains and accesses an entity graph within the database 920. Examples of functions and services supported by the social network server 922 include identifying other users in the messaging system 900 with whom a particular user has a relationship or who the particular user is "following", and also include identifying interests and other entities of a particular user.

[0091] The messaging client 902 may notify the user of the user device 328 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 902 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 902 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0092] Fig.10 1000 is a block diagram illustrating a software architecture 1004 that may be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware, such as a machine 1002 including a processor 1020, a memory 1026, and an I / O component 1038. In this example, the software architecture 1004 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1004 includes layers such as an operating system 1012, a library 1008, a framework 1010, and an application 1006. In operation, the application 1006 invokes an API call 1050 through the software stack and receives a message 1052 in response to the API call 1050.

[0093] The operating system 1012 manages hardware resources and provides public services. The operating system 1012 includes, for example, a core 1014, services 1016, and drivers 1022. The core 1014 serves as an abstraction layer between the hardware and other software layers. For example, the core 1014 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1016 can provide other public services to other software layers. Drivers 1022 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1022 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0094] The libraries 1008 provide low-level common infrastructure used by the applications 1006. The libraries 1008 may include system libraries 1018 (eg, C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1008 may include an API library 1024, such as a media library (e.g., a library for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting graphics content in two dimensions (2D) and three dimensions (3D) on a display, GLMotif for implementing a 3D user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1008 may also include various other libraries 1028 to provide many other APIs to the application 1006.

[0095] The framework 1010 provides a high-level common infrastructure used by the applications 1006. For example, the framework 1010 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 1010 can provide a wide range of other APIs that can be used by the applications 1006, some of which can be specific to a particular operating system or platform.

[0096] In an example, applications 1006 may include a home application 1036, a contacts application 1030, a browser application 1032, a book reader application 1034, a positioning application 1042, a media application 1044, a messaging application 1046, a game application 1048, and a variety of other applications such as third-party applications 1040. Applications 1006 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1006 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1040 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROIDTM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1040 can call the API call 1050 provided by the operating system 1012 to facilitate the functions described in this article.

[0097] Fig.11 The illustrative embodiment of the present invention is a diagrammatic representation of a machine 1100 or a computing device in which instructions 1110 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 1100 to perform any one or more of the methods discussed herein. For example, the instructions 1110 may cause the machine 1100 to perform any one or more of the methods described herein. The instructions 1110 transform a general-purpose, unprogrammed machine 1100 into a specific machine 1100 that is programmed to perform the functions described and illustrated in the manner described. The machine 1100 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1110 that specify actions to be taken by the machine 1100, either sequentially or otherwise. In addition, while a single machine 1100 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 1110 to perform any one or more of the methodologies discussed herein.

[0098] The machine 1100 may include a processor 1102, a memory 1104, and an I / O component 1106, which may be configured to communicate with each other via a bus 1144. In an example, the processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1108 that executes instructions 1110 and a processor 1112. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.11 Multiple processors 1102 are shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0099] The memory 1104 includes a main memory 1114, a static memory 1116, and a storage unit 1118, which can be accessed by the processor 1102 via the bus 1144. The main memory 1104, the static memory 1116, and the storage unit 1118 store instructions 1110 that implement any one or more of the methods or functions described herein. The instructions 1110 may also reside, in whole or in part, within the main memory 1114, within the static memory 1116, within the machine-readable medium 1120 within the storage unit 1118, within one or more of the processors 1102 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution of the instructions 1110 by the networked system 300.

[0100] I / O components 1106 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1106 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 1106 may include Fig.111106. In various examples, the I / O components 1106 may include output components 1128 and input components 1132. The output components 1128 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1132 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0101] In another example, the I / O component 1106 may include: a biometric component 1134, a motion component 1136, an environment component 1138, or a position component 1140, as well as various other components. For example, the biometric component 1134 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1136 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1138 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The location component 1140 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0102] A variety of technologies may be used to achieve communication. I / O components 1106 also include communication components 1142 that are operable to couple networked system 300 to network 1122 or device 1124 via coupling 1130 and coupling 1126, respectively. For example, communication components 1142 may include a network interface component or another suitable device that interfaces with network 1122. In other examples, communication components 1142 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Device 1124 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0103] In addition, the communication component 1142 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1142 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1142, such as a location obtained via Internet Protocol (IP) geolocation, ... Location obtained by signal triangulation, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0104] Various memories (e.g., memory 1104, main memory 1114, static memory 1116, and / or memory of processor 1102) and / or storage unit 1118 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1110) when executed by processor 1102 cause various operations to implement the disclosed examples.

[0105] The instructions 1110 may be transmitted or received over the network 1122 via a network interface device (e.g., a network interface component included in the communication component 1142) using a transmission medium and using any of a number of well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1110 may be transmitted or received to the device 1124 via a coupling 1126 (e.g., a peer-to-peer coupling) using a transmission medium.

[0106] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0107] A "user device" or "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices or client devices. A user device or client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0108] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0109] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by a temporary configuration (e.g., by software) or a permanent configuration to perform one or more processors of the related operation. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographic location (for example, in a home environment, an office environment, or a server cluster). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographic locations.

[0110] "Computer-readable media" or "machine storage media" refers to a single or multiple non-transitory storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media."

[0111] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0112] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0113] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method performed by a first mobile device for establishing a communication link between the first mobile device and a second mobile device, the method comprising: capturing, by the first mobile device, an image of the machine-readable code displayed on the second mobile device; decoding, by the first mobile device, communication settings information from the captured image of the machine-readable code; as well as A communication link is established with the second mobile device using the communication setup information.

2. The computer-implemented method of claim 1 , wherein: The communication setting information is dynamically generated.

3. The computer-implemented method of claim 1 , further comprising: A message is sent broadcasting at least a portion of the communication settings information.

4. The computer-implemented method of claim 1 , further comprising: Scanning is performed for a message from the second mobile device, the message broadcasting at least a portion of the communication setup information.

5. The computer-implemented method of claim 1 , further comprising: determining a transformation of a fiducial marker displayed by the second mobile device; receiving information specifying a posture of the second mobile device; determining a relative pose between the first mobile device and the second mobile device using the transformation and the information specifying the pose of the second mobile device; as well as A coordinate system is aligned between the first mobile device and the second mobile device.

6. The computer-implemented method of claim 5, wherein: The machine readable code is the fiducial mark.

7. The computer-implemented method of claim 6, wherein: The information specifying the posture of the second mobile device is encoded in the machine-readable code.

8. The computer-implemented method of claim 6, further comprising: receiving, via the communication link, an update to the information specifying a posture of the second mobile device; as well as A user interface action is performed on the first mobile device based on the received update to the information specifying the gesture of the second mobile device.

9. A non-transitory computer-readable storage medium comprising instructions that, when executed by a first mobile device, cause the first mobile device to perform operations for establishing a communication link between the first mobile device and a second mobile device, the operations comprising: capturing, by the first mobile device, an image of the machine-readable code displayed on the second mobile device; decoding, by the first mobile device, communication settings information from the captured image of the machine-readable code; as well as A communication link is established with the second mobile device using the communication setup information.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The communication setting information is dynamically generated.

11. The non-transitory computer-readable storage medium of claim 9, wherein: The operations also include: A message is sent broadcasting at least a portion of the communication settings information.

12. The non-transitory computer-readable storage medium of claim 9, wherein: The operations also include: determining a transformation of a fiducial marker displayed by the second mobile device; receiving information specifying a posture of the second mobile device; determining a relative pose between the first mobile device and the second mobile device using the transformation and the information specifying the pose of the second mobile device; and A coordinate system is aligned between the first mobile device and the second mobile device.

13. The non-transitory computer readable storage medium of claim 12, wherein: The machine readable code is the fiducial mark.

14. The non-transitory computer-readable storage medium of claim 12, wherein: The operations also include: receiving, via the communication link, an update to the information specifying a posture of the second mobile device; and A user interface action is performed on the first mobile device based on the received update to the information specifying the gesture of the second mobile device.

15. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for establishing a communication link between a first mobile device and a second mobile device, the operations comprising: capturing, by the first mobile device, an image of the machine-readable code displayed on the second mobile device; decoding, by the first mobile device, communication settings information from the captured image of the machine-readable code; and A communication link is established with the second mobile device using the communication setup information.

16. The computing device of claim 15, wherein: The communication setting information is dynamically generated.

17. The computing device of claim 15, wherein: The operations also include: Scanning is performed for a message from the second mobile device broadcasting at least a portion of the communication settings information.

18. The computing device of claim 15, wherein: The operations also include: determining a transformation of a fiducial marker displayed by the second mobile device; receiving information specifying a posture of the second mobile device; determining a relative pose between the first mobile device and the second mobile device using the transformation and the information specifying the pose of the second mobile device; and A coordinate system is aligned between the first mobile device and the second mobile device.

19. The computing device of claim 18, wherein: The machine readable code is the fiducial mark.

20. The computing device of claim 19, wherein: The operations also include: receiving, via the communication link, an update to the information specifying a posture of the second mobile device; and A user interface action is performed on the first mobile device based on the received update to the information specifying the gesture of the second mobile device.

Images