Device communication by high frequency light coding

Abstract

Embodiments set forth herein set forth a technique for enabling a computing device to securely communicate with a peripheral computing device. According to some embodiments, the method can include the steps of: (1) receiving, at a detection sensor of the computing device, an optical signal from the peripheral computing device, wherein the optical signal is received at a first frequency that is higher than a second frequency that is capable of being detected by a camera system of the computing device; (2) extracting information from the optical signal; and (3) performing an operation using the information.

Description

Technical Field

[0001] The described implementation scheme generally relates to computing devices configured to communicate with each other. More specifically, the described implementation scheme relates to enabling privacy-aware, proximity-based wireless communication between a computing device and a peripheral computing device by using a unidirectional, high-frequency optical signal transmitted by a peripheral computing device and detecting the optical signal using a detection sensor of the computing device. Background Technology

[0002] In recent years, the average number and types of personal peripheral computing devices have surged. For example, it's common for individuals to own wearable devices (e.g., fitness trackers), headphones, set-top boxes, wireless speakers, and so on. Notably, owning these peripheral computing devices can provide a rich user experience, as each device offers specialized functionality to meet a given user's needs throughout the day. Furthermore, an increasing number of peripheral computing devices are being implemented as Internet of Things (IoT) devices capable of communicating with other computing devices. Many of these peripheral computing devices include network circuitry that enables wireless communication with other computing devices (e.g., via Wi-Fi or Bluetooth). However, such wireless communication can be intercepted by malicious computing devices and may compromise sensitive user data. Moreover, many of these peripheral computing devices lack a user interface that allows users to determine why the device is malfunctioning (e.g., speakers, wearables, etc.). This lack of a user interface also makes it difficult to effectively configure complex settings for the device (e.g., Wi-Fi passwords, user account information, etc.), for example, when they require an "out-of-the-box" setup process. In some cases, more advanced computing devices (e.g., smartphones, tablets, etc.) can be paired with peripheral computing devices and the setup process can be automated to assist the peripheral computing device. Unfortunately, this approach is prone to unresolved security issues. For example, a nearby malicious computing device could emulate the peripheral computing device and trick the user into pairing with or sending sensitive information to the malicious device. Summary of the Invention

[0003] To address the aforementioned shortcomings, the representative implementation described herein discloses various techniques for privacy-aware methods that enable proximity-based wireless communication between computing devices and peripheral computing devices using unidirectional, high-frequency optical signals.

[0004] According to some embodiments, a method for enabling a computing device to securely communicate with a peripheral computing device is disclosed. According to some embodiments, the method may include the steps of: (1) receiving an optical signal from the peripheral computing device at a detection sensor of the computing device, wherein the optical signal is received at a first frequency higher than a second frequency detectable by a camera system of the computing device; (2) extracting information from the optical signal; and (3) performing an operation using the information.

[0005] According to some embodiments, another method for enabling a peripheral computing device to communicate securely with a computing device is disclosed. According to some embodiments, the method may include the following steps at the peripheral computing device: (1) encoding information using analog modulation and coding techniques; (2) transmitting the information in the optical signal to a detection sensor of the computing device at a first frequency higher than a second frequency detectable by a camera system of the computing device; and (3) performing an operation.

[0006] Other embodiments include a non-transitory computer-readable storage medium configured to store instructions that, when executed by a processor included in a computing device, cause the computing device to perform steps of any of the methods described above. Further embodiments include a computing device configured to perform various steps of any of the methods described above.

[0007] Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings which illustrate the principles of the described embodiments by way of example. Attached Figure Description

[0008] This disclosure will be more readily understood from the following detailed description taken in conjunction with the accompanying drawings, wherein similar reference numerals denote similar structural elements.

[0009] Figure 1 Block diagrams are shown of different computing devices that can be configured to implement different aspects of the various technologies described herein, according to some implementation schemes.

[0010] Figures 2A to 2C A conceptual diagram is shown of an exemplary scenario in which computing devices and peripheral computing devices communicate via high-frequency optical coding, according to some implementation schemes.

[0011] Figure 3 A method for enabling a computing device to communicate securely with peripheral computing devices, according to some embodiments, is shown.

[0012] Figure 4 A method for enabling a peripheral computing device to communicate securely with a computing device, according to some embodiments, is shown.

[0013] Figure 5A conceptual diagram of an exemplary user interface that can be implemented at a computing device participating in wireless communication with a peripheral computing device, according to some implementation schemes, is shown.

[0014] Figure 6 Detailed views of a computing device according to some embodiments are shown, which may represent the various technologies described herein. Figure 1 Computing devices. Detailed Implementation

[0015] Representative applications of the apparatus and methods according to embodiments of the present invention are provided in this section. These examples are provided only to add context and aid in understanding the described embodiments. It will therefore be apparent to those skilled in the art that the embodiments currently described can be practiced without some or all of these specific details. In other instances, well-known process steps have not been described in detail to avoid unnecessarily obscuring the embodiments currently described. Other applications are possible, such that the following examples should not be considered limiting. It should be noted that the terms "frequency" and "carrier frequency" are used interchangeably herein.

[0016] The embodiments described herein illustrate privacy-aware, proximity-based wireless communication between a computing device and a peripheral computing device by using a unidirectional, high-frequency optical signal transmitted by a peripheral computing device and detecting the optical signal using a detection sensor of the computing device. The optical signal may be transmitted by a light source (e.g., a light-emitting diode (LED)) of the peripheral computing device and detected by a photodiode or other suitable detection sensor of the computing device. The computing device may be a smartphone, tablet, laptop, etc. The peripheral computing device may be wearable (e.g., a smartwatch, headphones, etc.), an Internet of Things (IoT) device (e.g., a light bulb, appliance, speaker, security system, thermostat, smoke detector, smart hub, etc.), an automobile, a machine, or any suitable device capable of using the device's light source to transmit information at a high frequency.

[0017] According to some implementations, communication is intended to be proximity-based. In some implementations, the limiting factor for operation is range; for example, the detection sensors of the computing device must be within a certain range (e.g., a few inches, a few feet, etc.) of the light source in order to receive the light signal. In some implementations, the limiting factor for operation is line of sight; for example, the detection sensors and the light source must have a clear path without any obstacles (walls, barriers, etc.), but there is no specific range limitation. Proximity-based interaction achieves privacy by preventing (e.g., in different rooms) other malicious computing devices from intercepting the light signal. Furthermore, proximity-based interaction enhances privacy by providing the computing device with intentional gestures for interacting with peripheral computing devices by bringing the two computing devices within a specific range of each other.

[0018] Peripheral computing devices can be configured to remove the direct current (DC) element from optical signals using a baseline or other suitable technique. Because the DC element is removed, the disclosed implementation remains invariant to light intensity and the presence of sunlight. By transmitting bits in the optical signal at a frequency much higher than ambient light frequency, interference from artificial ambient light is unlikely. Therefore, the disclosed technique provides robust wireless communication to environmental factors. The bit rate also enables privacy protection, as another device cannot intercept the signal through a window or from a distance. Since the bit rate (e.g., 3200 bps) is higher than the camera's sampling rate, the optical signal is a short-throw that some cameras, including those in the computing device, cannot detect.

[0019] Various encoding / decoding methods can be used by peripheral computing devices and computing devices. Encoding / decoding methods may include time-domain methods, frequency-shift keying (FSK), forward error correction (FEP), phase shift, amplitude modulation, etc. For example, some implementations may use time-domain methods that encode 1s and 0s by switching a light source on and off at a constant frequency. In some implementations, FSD may be used to transmit each bit (or symbol) at different frequencies using an optical signal while maintaining a constant time interval between each bit (or symbol).

[0020] In some implementations, digital packets may comprise a small number of bits (e.g., 36 bits (9 hexadecimal digits)). According to some implementations, the optical signal may be unidirectional and asynchronous, as the computing device may not transmit a response to peripheral computing devices. Therefore, in some implementations, forward error correction may be used to detect and potentially correct errors at the computing device. Additionally or alternatively, some implementations implement a configurable retransmission variable that can be tuned by the user to enable the configuration of a minimum number of retransmissions to ensure digital packets arrive without wasting resources by transmitting digital packets too many times in the optical signal. In some implementations, a frame (e.g., a 10-bit frame) is transmitted from the light source to the detection sensor before transmitting a digital packet including information indicating the start of the digital packet. Other packet sizes and other frame sizes are also possible.

[0021] According to some embodiments, the detection sensor can be a multi-purpose sensor. When capturing video and / or images using a camera on a computing device (e.g., a smartphone), the detection sensor can compensate for flicker in ambient light. The detection sensor can be configured to receive high-frequency light signals. In some embodiments, the detection sensor may be mounted as part of the strobe or flash module of the camera system on the computing device. Alternatively, the detection sensor may be included as a general-purpose component of the computing device itself or any other component of the computing device.

[0022] As described herein, the disclosed techniques may relate to using a detection sensor to receive high-frequency optical signals (generated by an LED of a peripheral computing device) encoded with information. According to some embodiments, the detection sensor characteristics may include (i) multiple channels defined by the sensitivity of each channel to light of various wavelengths (colors); (ii) channels sensitive to invisible light (including infrared light); (iii) simultaneous use of multiple channels for auxiliary purposes, such as distinguishing indoor and outdoor lighting conditions; and (iv) the detection sensor may also be characterized by its sampling rate (e.g., 16 kHz or 48 kHz, etc.).

[0023] Wireless communication using high-frequency optical signals can be used in a variety of applications, such as pairing computing devices, Quick Response Code (QR) identification, communicating the status of peripheral devices to computing devices, conveying errors or warnings to help diagnose peripheral computing devices, or a combination thereof. For example, traditional warning lights are abstract. A "Check Engine" indicator in a car may indicate a problem with the engine, but the user is not informed what the problem is or how to resolve it. According to some implementations, a user can bring a computing device (e.g., a smartphone) within physical range of a peripheral computing device that emits an optical signal and a notification that includes a description of the warning and how to resolve the problem associated with the warning in real-time or near real-time. Real-time or near real-time can refer to a timeframe of two seconds or less.

[0024] A more detailed discussion of these technologies is presented below and combined with Figure 1 , Figures 2A to 2C and Figures 3 to 6 The description includes detailed illustrations of systems and methods that can be used to implement these technologies.

[0025] Figure 1 A block diagram 100 is shown illustrating various computing devices 102 that can be configured to implement aspects of the techniques described herein, according to some embodiments. Specifically, Figure 1 A high-level overview of computing device 102 is shown, as illustrated, which may include at least one processor 104, at least one memory 106, and at least one storage device 132. According to some embodiments, processor 104 can be configured to work in conjunction with memory 106 and storage device 132 to enable computing device 102 to implement various technologies set forth in this disclosure. According to some embodiments, storage device 132 can represent a storage device entity accessible to computing device 102, such as a hard disk drive, solid-state drive, mass storage device, remote storage device, storage service, etc. For example, storage device 132 may be configured to store an operating system (OS) file system volume 134 that can be mounted on computing device 102, wherein operating system (OS) file system volume 134 includes an OS 108 compatible with computing device 102.

[0026] According to some implementation schemes, and such Figure 1 As shown, OS 108 enables signal manager 110 to execute on computing device 102. It should be understood that OS 108 also enables various other processes to execute on computing device 102 (e.g., OS daemons, native OS applications, user applications, etc.). According to some embodiments, signal manager 110 on computing device 102 can be configured to interact with signal manager 110 executing on peripheral computing device 102 to perform the techniques described herein.

[0027] like Figure 1 As shown, and as described in more detail herein, OS 108 can be configured to enable signal manager 110 to interact with various communication components 116 included in / available in computing device 102. Communication components 116 may include, for example, a Wi-Fi interface 118, a near-field communication (NFC) interface 120, a Bluetooth interface 122, at least one speaker 124, at least one microphone 126, at least one camera system 128, and at least one light source (e.g., a light-emitting diode (LED), a laser, etc.). Figure 1 LED 130 is shown. Communication component 116 may also include... Figure 1 Components not shown include, for example, cellular interfaces, Ethernet interfaces, display interfaces, input interfaces (e.g., buttons, touch surfaces, dials, etc.). It should be noted that these examples are not intended to represent an exhaustive list in any way, and any form of communication interface may be included in communication component 116. For example, communication component 116 may include a Global Positioning System (GPS) interface that allows computing devices 102 to recognize when they are close to each other. This, for example, can provide an additional level of security related to recognizing when a user intends to use their computing device 102 to participate in the processes described herein with other computing devices 102.

[0028] According to some embodiments, camera system 128 may be able to acquire images and / or video, for example, using at least one lens system, at least one imaging sensor, etc. According to some embodiments, camera system 128 may include at least one detection sensor 131. Alternatively, as previously described herein, detection sensor 131 may be included as a component of computing device 102 (or any other component / subcomponent of computing device 102) without departing from the scope of this disclosure.

[0029] According to some embodiments, the detection sensor 131 may be mounted in a portion of the strobe / flash module included in the camera system 128, or may be mounted in any other portion of the camera system 128. The detection sensor 131 may be partially or completely hidden or otherwise obscured and invisible. In one embodiment, the detection sensor 131 may be a photodiode. The detection sensor 131 may be configured to prevent ambient light frequencies from producing visible artifacts in images and / or videos captured by the camera system 128. According to some embodiments, the detection sensor 131 may be configured for a variety of purposes. For example, in addition to compensating for any flicker in ambient light when capturing video, the detection sensor 131 may be configured to receive high-frequency light signals emitted from LED 130 of the peripheral computing device 102.

[0030] As described herein, the detection sensor 131 of computing device 102 can be used for proximity-based interaction with peripheral computing device 102. Using an encoded high-frequency light signal transmitted by LED 130 of peripheral computing device 102 and detected by detection sensor 131, data privacy can be maintained due to the short transmission range of the light signal and its dissipation during its travel; or through the line-of-sight path of the light signal. This prevents malicious remote computing devices from stealing the light signal because the light signal does not travel through walls.

[0031] The detection sensor 131 is capable of detecting optical signals using at least two channels: (1) an infrared (IR) channel, and (2) an IR and visible light channel, which enables the differentiation between indoor and outdoor settings. The detection sensor 131 is capable of distinguishing individual colors of visible light and can provide a separate channel for each color. The detection sensor 131 is capable of providing a sampling rate in the range of 16 kHz to 48 kHz. Furthermore, the detection sensor 131 is capable of detecting frequencies in the range of 50 Hz to 4 kHz.

[0032] According to some implementations, the light signal emitted by LED 130 cannot be detected by camera system 128 because the light signal is modulated at a frequency higher than that detectable by camera system 128. Detection sensor 131 can be configured to detect the light signal because it is capable of receiving light signals at high frequencies (e.g., modulated at at least 3200 Hz). Therefore, the disclosed technology enables privacy-preserving, short-range, proximity-based wireless communication between computing device 102 and peripheral computing device 102. The light signal may not be able to be read through windows, walls, or from a distance (e.g., several meters) from LED 130. Furthermore, detection sensor 131 uses less power than camera system 128, and therefore using detection sensor 131 to receive light signals conserves battery life of computing device 102. The disclosed technology does not require additional hardware to be added to peripheral computing device 102, which already includes LED 130.

[0033] To further enhance privacy and security, the optical signal can be paired with auxiliary data. For example, because the optical signal travels a long distance, the computing device 102 must be placed close to the peripheral computing device 102 that emitted the optical signal. By moving the computing device 102 closer to the peripheral computing device 102, a notification can be received or generated by the computing device 102 when the optical signal is received. This notification can instruct the user to make an intentional gesture by moving the computing device 102 closer to the peripheral computing device 102. Therefore, when a notification is received or generated, the computing device 102 can pair the notification with the optical signal and present a specific user interface on the computing device based on information extracted from the optical signal. Such a technique ensures that the user makes an intentional gesture and is close enough to the peripheral computing device 102 to achieve data security and privacy. Furthermore, such a technique prevents the user interface from being unintentionally presented on the computing device 102 at an unexpected time. Presenting the user interface when an intentional gesture is detected saves computing resources by preventing the user interface from being presented when no intentional gesture is present.

[0034] According to some implementations, the signal manager 110 on computing device 102 can act as a "configurator" for the signal manager 110 executed on peripheral computing device 102, for example, when the signal manager 110 on peripheral computing device 102 broadcasts a setup request to a nearby computing device 10 via a light signal emitted from LED 130 on peripheral computing device 102. The signal manager 110 on computing device 102 can accept the setup request. Subsequently, and after performing secure pairing technology, the signal manager 110 (on computing device 102) can access and transmit various information associated with computing device 102 to the signal manager 110 (on peripheral computing device 102) for installation on peripheral computing device 102. This information may include, for example, Wi-Fi information (e.g., Service Set Identifier (SSID) / password / encryption key), user account information (e.g., cloud account login / password / encryption key), encryption key sets, etc., which enables peripheral computing device 102 to provide a rich user experience. It should be noted that the foregoing examples are not intended to represent an exhaustive list in any way, and information of any kind may be shared among computing devices 102 where appropriate.

[0035] According to some embodiments, the signal manager 110 on the computing device 102 can act as a "scanner" by receiving light signals from the LED 130 of the peripheral computing device 102, decoding the light signals, and identifying a Quick Response (QR) code included in the payload of the light signals. According to some embodiments, the signal manager 110 on the computing device 102 can act as a "trouble finder" for the peripheral computing device 102. For example, the light signals can be encoded with a payload containing information related to the status of the peripheral computing device 102, errors or warnings related to the peripheral computing device 102, or some combination thereof. The LED 130 of the peripheral computing device 102 can transmit light signals at a high frequency by flashing on and off at a high rate to transmit bits at a high rate. The light signals emitted by the peripheral computing device 102 can be received by the detection sensor 131 of the computing device 102. The signal manager 110 of the computing device 102 can decode the light signals to extract information. In response to the extracted information, the signal manager 110 of the computing device 102 can perform operations based on that information. This information can be presented in the user interface of the computing device 102. In some cases, this information may indicate that the wireless connection link (e.g., Wi-Fi) has been disconnected, and the computing device 102 may attempt to re-establish the wireless connection link for the peripheral computing device 102.

[0036] In some implementations, this information may include the model of the peripheral computing device 102, the protocol used by the peripheral computing device 102, the Internet Protocol (IP) address of the peripheral computing device 102, a unique identifier for the peripheral computing device 102, and the name of the peripheral computing device 102. In some implementations, operations performed by the computing device 102 may be combined with operations performed by the peripheral computing device 102. For example, this operation may be a pairing operation, wherein each of the computing device 102 and the peripheral computing device 102 performs a "handshake" by exchanging information about each device with the other corresponding device and transmitting one or more confirmation messages along with a private key and / or a public key to the other corresponding device. In some implementations, operations performed by the computing device 102 may include setting up the peripheral computing device 102 by configuring one or more parameters of the peripheral computing device 102. In some implementations, operations performed by the computing device 102 may include adding the peripheral computing device 102 to a device group managed by a user account.

[0037] therefore, Figure 1 A high-level overview of the various components / entities that may be included in each computing device 102 is shown to enable the embodiments described herein to be correctly implemented. As described in more detail below, these components / entities can be utilized to provide privacy-aware wireless communication between computing device 102 and peripheral computing devices 102 using high-frequency optical coding, thereby enhancing overall security.

[0038] Figures 2A to 2C A conceptual diagram of an exemplary scenario is shown, according to some embodiments, in which a peripheral computing device 102-2 transmits an optical signal to a nearby computing device 102-2 to perform an operation. According to some embodiments, the operation may include setting up the peripheral computing device 102-2, reconfiguring the peripheral computing device 102-2, establishing a new pairing with the peripheral computing device 102-2, presenting a notification (e.g., describing the type of error, status, or warning related to the peripheral computing device 102-2) on the user interface of the computing device 102-1, causing a change in the status or parameters of the peripheral computing device 102-2 (e.g., re-establishing a wireless communication link, restarting, releasing memory, etc.).

[0039] like Figure 2AAs shown, the first step 210 may involve the peripheral computing device 102-2 transmitting an optical signal encoded in packet 212. According to some embodiments, the optical signal may be encoded and transmitted by LED 130-2 of the peripheral computing device 102-2. The optical signal includes packet 212 (e.g., digital packets) transmitted at a high frequency (e.g., at least 3200 Hz). The optical signal can be encoded by removing DC elements from it. The optical signal can be encoded via a time-domain method, wherein 1s and 0s are transmitted by turning LED 130-2 on and off at a constant frequency. The optical signal can be encoded via frequency shift keying (FSK) by transmitting each bit or symbol at different frequencies while keeping the time interval between each bit or symbol constant. Furthermore, in some embodiments, amplitude modulation may be employed when encoding the optical signal.

[0040] Packet 212 may include a number of bits (e.g., 36 bits). Additionally, packet 212 may include a start frame 213, a payload 214, and / or error correction 216. The start frame 213 may include a number of bits (e.g., 10, 15, 20) in a frame transmitted by LED 130 in an optical signal before the transmission of payload 214 and / or error correction 216. The start frame 213 may indicate the start of payload 214 and / or error correction 216. In some embodiments, payload 214 may be included in bits 0 through 31, and error correction 216 may be included in bits following the payload. In some embodiments, error correction 216 may be included in the same bit range as payload 214. Payload 214 may include any suitable information, such as a message, protocol, device identifier, model, status code, warning code, error code, programming instruction, request, or some combination thereof. Error correction 216 may include codes that enable the detection and correction of errors in packet 212. Error correction 216 can use forward error correction (e.g., GoLay code) to provide robustness against bit-by-bit errors. The decoder 221 of computing device 102-1 can receive bits transmitted in packet 212 from the optical signal, determine whether there are any errors in the code transmitted in packet 212, and correct and retrieve the original code included in packet 212 (e.g., up to 3 bits of error per 12 bits).

[0041] According to some implementations, the setup mode can be announced via packet 212, which is emitted via LED 130-2 included in the peripheral computing device 102-2. Payload 214 may include a device identifier (ID) (e.g., hardware / software identifier) ​​associated with the peripheral computing device 102-2, an indication of a setup request (e.g., a predefined code, message, etc.), status codes (e.g., software status, processor status, memory status, network status, etc.), error codes, warning codes, etc. In this way, a nearby computing device 102-1 can receive (e.g., using their respective detection sensors 131) setup mode announcements from other nearby computing devices 102 (e.g., peripheral computing devices 102-2) and respond as appropriate. It should be noted that a nearby computing device 102 can be configured to receive a light signal when an intentional gesture is detected by moving the computing device 102-1 into physical proximity to the peripheral computing device 102-2.

[0042] like Figure 2B As shown, step 220 may involve computing device 102-1 receiving an optical signal including packet 212 and extracting information from packet 212 using decoder 221 of signal manager 110-1. The optical signal may be received at computing device 102-1 at detection sensor 131. Decoder 221 may decode packet 212 based on a method used to encode packet 212 in the optical signal.

[0043] Next, in Figure 2C At step 230, computing device 102-1 may perform operations that may include peripheral computing device 102-2. For example, if the information extracted from the optical signal is a pairing request from peripheral computing device 102-2, the pairing request may be presented on the user interface of computing device 102-1. If the user selects the option to accept or approve the pairing request, computing device 102-1 may transmit acceptance and information about computing device 102-1 to peripheral computing device 102-2 to enable pairing between peripheral computing device 102-2 and computing device 102-1. In some embodiments, the operation may not include peripheral computing device 102-2. For example, the information extracted from the optical signal may include a low battery warning indicating that peripheral computing device 102-2 has low battery, and this indication may be presented on the user interface of computing device 102-1. In other words, the information extracted from the optical signal may cause the computing device to perform operations, such as presenting a notification describing the information, without relaying it back to peripheral computing device 102-2. In some implementations, the information extracted from the optical signal can enable the computing device 102-1 to configure, set, pair, repair, etc., the parameters or aspects of the peripheral computing device 102-2.

[0044] like Figure 2CAs illustrated in the example, the information extracted from the optical signal includes a configuration request from the peripheral computing device 102-2. After receiving the optical signal at the detection sensor 131, the computing device 102-1 extracts the information encoded in the optical signal and performs an operation based on the configuration request represented in that information. The computing device 102-1 can transmit configuration information 262 to the Bluetooth interface 122-2 of the peripheral computing device 102-2 via a secure communication link 252 using the Bluetooth interface 122-1. According to some embodiments, the secure communication link 252 can be established using the corresponding Bluetooth interface 122 and can be based on cryptography. For example, the cryptography can be a symmetric encryption key that can be used to form the secure communication link 252 (and to protect various packets sent through the secure communication link). Alternatively, the cryptography can provide a basis for establishing at least one encryption key that can be used to form the secure communication link. For example, each of peripheral computing device 102-2 and computing device 102-1 has access to an encryption algorithm that processes cipher 231 to derive at least one symmetric key, through which secure communication link 252 is established. In this way, even if a malicious party somehow gains access to the cipher during the pairing process, they cannot derive the at least one symmetric key that enables secure communication link 252 to be established.

[0045] Configuration information 262 may include, for example, Wi-Fi information (e.g., Service Set Identifier (SSID) / password / encryption key), user account information (e.g., cloud account login / password / encryption key), encryption key sets, etc., which enables peripheral computing device 102-2 to provide a rich user experience. Peripheral computing device 102-2 may then process configuration information 262 as appropriate. For example, peripheral computing device 102-2 may use the Wi-Fi information included in configuration information 262 to connect to a corresponding Wi-Fi network. Alternatively, peripheral computing device 102-2 may use the user account information included in configuration information 262 to access data / services provided by a cloud service. In yet another example, peripheral computing device 102-2 may use the encryption key set to enter a trust loop with other computing devices 102 and provide various functionalities. It should be noted that the foregoing examples are merely exemplary, and any form of information may be included in configuration information 262. Furthermore, peripheral computing device 102-2 may process configuration information 262 in any suitable manner, enabling peripheral computing device 102-2 to employ various functionalities.

[0046] therefore, Figures 2A to 2C A conceptual diagram illustrating an example scenario where computing device 102-1 wirelessly communicates with peripheral computing device 102-2 via short-range, proximity-based high-frequency optical coding to perform various operations, according to some implementation schemes. Next, Figures 3 to 5(As described in more detail below) provides an additional high-level decomposition of the techniques described herein.

[0047] Figure 3 A method 300, according to some embodiments, is shown for enabling a computing device 102 (e.g., computing device 102-1) to securely communicate with a peripheral computing device 102 (e.g., peripheral computing device 102-2). This method may be implemented by instructions stored on a memory device of the computing device 102-1 and executed by a processing device communicatively coupled to the memory device.

[0048] As shown in the figure, method 300 begins at step 302, in which computing device 102-1 receives an optical signal from peripheral computing device 102-2 at the detection sensor 131 (e.g., as described above). Figures 2A to 2B The optical signal can be received at a first frequency higher than the second frequency that can be detected by the camera system 128. For example, in some embodiments, the received optical signal can be modulated at a frequency of at least 3200 Hz, which is higher than the maximum frequency of the signal that the camera system 128 can receive or detect (e.g., the second frequency). In some embodiments, the first frequency is in the range of 50 Hz to 4 kHz. In some embodiments, the first frequency can be in any suitable range that can be detected by the detection sensor 131 but not by the camera system 128. In some embodiments, the optical signal is generated by at least one light source (LED 130) included on the peripheral computing device 102-2. Furthermore, the optical signal can be received from the peripheral computing device 102-1 as asymmetric, unidirectional communication.

[0049] In some implementations, computing device 102-1 may use detection sensor 131 to filter ambient light to prevent interference with data captured by camera system 128. Thus, detection sensor 131 can be used for a variety of technical purposes.

[0050] At step 304, in response to receiving the optical signal, computing device 102-1 extracts information from the optical signal (e.g., as described above in conjunction with...). Figure 2BThe information may include any suitable information, such as the model of the peripheral computing device 102-2, the name of the peripheral computing device 102-2, the IP address of the peripheral computing device 102-2, the protocol used by the peripheral computing device 102-2 to send status codes, error codes, warning codes, etc., messages (e.g., error codes, status codes, warning codes, etc.), programming instructions, requests, etc. In some embodiments, the information may include an identifier of a protocol and a code defined by that protocol (e.g., an error code), and the computing device 102-1 may determine the operation to be performed based on the protocol and the code. For example, the computing device 102-1 may use a lookup table stored in a database to identify a protocol with an identifier and translate the code by recognizing the corresponding error description as a code. In some embodiments, the information may include programming instructions executed by the computing device 102-1 to perform one or more operations. The programming instructions may be implemented by an executable program or script.

[0051] This information can be encoded in digital packets of the optical signal by the peripheral computing device 102-2 using analog modulation and coding techniques such as time-domain, frequency-shift keying, or some combination thereof. Alternatively, amplitude modulation, forward error correction coding, or some combination thereof can be used to encode the information in the digital packets. Thus, the digital packets can include error-correction sections used by the computing device 102-1 to correct any errors detected in the information included in the digital packets. In some embodiments, the digital packets can encode information in the payload, represented by bits 0 to 31. However, any suitable number of bits capable of performing the techniques described herein can be used in the digital packets. This information can be extracted by the computing device 102-1 performing decoding on the optical signal.

[0052] In some implementations, computing device 102-1 can simultaneously receive a second optical signal at a second frequency, which is perceptible to humans. That is, the light source can simultaneously generate and transmit two optical signals. One optical signal can be transmitted simultaneously with the human-perceptible second signal as a first signal that is imperceptible to humans. The second signal allows the user to determine whether there is a problem with peripheral computing device 102-2 or whether peripheral computing device 102-2 is attempting to communicate and allows the user to bring computing device 102-1 within physical proximity of peripheral computing device 102-2 to receive the first optical signal at the imperceptible first signal.

[0053] In step 306, in response to extracting information from the optical signal, computing device 102-1 uses the information to perform an operation (e.g., as described above in conjunction with...). Figure 2CThe operation may include pairing computing device 102-1 and peripheral computing device 102-2, identifying QR codes, identifying the status of peripheral computing device 102-2 (wireless connection loss, processing power, memory, battery usage or remaining level, etc.), detecting errors or warnings (e.g., peripheral computing device 102-2 overheating), or some combination thereof.

[0054] According to some embodiments, computing device 102-1 may receive (or generate) an operation-related notification before receiving the optical signal. This notification may represent an intention gesture triggered by movement of the computing device within physical proximity to peripheral computing device 102-2. Computing device 102-1 may determine whether the notification and optical signal are received within a threshold time period (e.g., concurrently, synchronously, simultaneously, within less than 1 second, 2 seconds, etc.). The threshold time period can be configured and used to further ensure that the intention gesture and optical signal are received within approximately the same time period to enhance security and privacy. In response to determining that the notification and optical signal are received within the threshold time period, computing device 102-1 may pair the notification and optical signal. In response to this pairing, computing device 102-1 may present an information-related notification on computing device 102-1. In some embodiments, the notification may include a description of the operation to be performed and provide an option to approve the operation. In some embodiments, computing device 102-1 may receive a selection of the option to approve the operation. In some implementations, in response to receiving a selection of an option to approve the operation, computing device 102-1 may indicate approval of the operation to peripheral computing device 102-2. For example, the operation may include a pairing process between computing device 102-1 and peripheral computing device 102-2, and the optical signal may include a request from peripheral computing device 102-2 to pair with computing device 102-1.

[0055] It should be noted that the above technologies can be combined to implement additional steps to achieve extended functionality for communication between computing device 102-1 and peripheral computing device 102-2.

[0056] For example, considering a scenario where peripheral computing device 102-2 is an audio component configured to play back audio from multiple sources (e.g., music services, pairing devices, etc.), method 300 may also involve prompting (to computing device 102-1) the user with login information associated with the music service the user has registered for. Additionally, computing device 102-1 may offer trial opportunities for various online services available to and associated with computing device 102-1 / peripheral computing device 102-2, especially when the user cannot provide login for the aforementioned music service. For example, computing device 102-1 may identify the type of peripheral computing device 102-2 and then interact with online services to identify any available free trials associated with the purchase of peripheral computing device 102-2.

[0057] In some cases, computing device 102-1 may be designed to participate as a component within a smart home environment. In this scenario, computing device 102-1 may prompt the user with different smart home configuration settings that should be applied to peripheral computing device 102-2. For example, computing device 102-1 may be configured to (1) present at least one home previously set by the user (e.g., “Cupertino home”), (2) present options for creating (i.e., establishing) at least one home when the user has not yet set up a home, and so on. In any case, when the user selects a home, computing device 102-1 may be configured to (1) present at least one room previously set by the user (e.g., “living room”), (2) present options for creating (i.e., establishing) at least one room when the user (e.g., within the selected home) has not yet set up a room, and so on. It should be noted that the foregoing examples are exemplary, and any smart home attributes may be assigned to peripheral computing device 102-1 during the operations or processes described herein. For example, the peripheral computing device 102-2 may be configured to participate as a separate speaker within a particular speaker group (e.g., belonging to a house and / or room). As another example, the peripheral computing device 102-2 may be configured as one of two speakers in a stereo configuration or one of many speakers in a surround sound configuration. In yet another example, the speaker may be configured as a single speaker that typically roams throughout a particular home.

[0058] therefore, Figure 3 Methods for enabling computing device 102 to participate in secure wireless communication with peripheral computing device 102 using high-frequency optical coding are described. Advantageously, the operations provided herein range from simple scenarios (e.g., sharing Wi-Fi information, presenting errors, statuses, warnings, etc.) to more complex scenarios (e.g., smart home configuration). In any case, the computing device 102 described herein can be configured to provide user interfaces that guide the user through various operations associated with peripheral computing device 102. Subsequently, peripheral computing device 102 can be configured to interact with and provide various levels of feedback (e.g., operability indications, confirmation indications, etc.), which will be discussed below in conjunction with... Figure 4 Describe its details.

[0059] Figure 4 A method 400 for performing a setup process at a peripheral computing device 102 (e.g., peripheral computing device 102-1) according to some embodiments is shown. This method may be implemented by instructions stored on a memory device of the peripheral computing device 102-2 and executed by a processing device communicatively coupled to the memory device.

[0060] As shown in the figure, method 400 begins at step 402, in which peripheral computing device 102-2 encodes information using analog modulation and coding techniques (e.g., time-domain, phase-shift keying, etc.). At step 404, peripheral computing device 102-2 transmits information in the optical signal to the detection sensor 131 of computing device 102-1 at a first frequency higher than the second frequency detectable by camera system 128. This information may include the model number of peripheral computing device 102-2, the name of peripheral computing device 102-2, the IP address of peripheral computing device 102-2, a request to pair peripheral computing device 102-2 with computing device 102-1, the protocol used by peripheral computing device 102-2, a quick response (QR) code, the status of peripheral computing device 102-2, errors or warnings associated with peripheral computing device 102-2, or a combination thereof. At step 406, in response to the information in the transmitted optical signal, peripheral computing device 102-2 can perform operations as described herein.

[0061] in addition, Figure 5 A conceptual diagram 500 of an exemplary user interface, according to some embodiments, is shown at a computing device (e.g., computing device 102-1) that participates in secure and privacy-preserving wireless communication with a peripheral computing device 102 (e.g., peripheral computing device 102-2). The user interface may be presented upon detection of an intention gesture. The intention gesture may be determined by the user moving the computing device 102-1 within physical proximity to the peripheral computing device 102-2. Figure 5 As shown, computing device 102-1 (specifically, signal manager 110-1 executed on computing device 102-1) can receive optical signals from peripheral computing device 102-2 via detection sensor 131. Then, computing device 102-2 can extract information from the optical signals and perform operations. Based on the information included in the optical signals, signal manager 110-1 can perform a certain operation.

[0062] In one example, where the information includes a setup request, the operation may include presenting a user interface 502 that informs the user of computing device 102-1 about a setup request issued by peripheral computing device 102-2. For example, user interface 502 may include images, animations, sounds, etc., that draw attention to computing device 102-1 and prompt the user whether he or she wants to use computing device 102-1 to pair / configure peripheral computing device 102-2. Figure 5 As shown, the user interface 502 may include, for example, a visual representation of the peripheral computing device 102-2 based on the device ID transmitted by the peripheral computing device 102-2 (e.g., as described above in conjunction with...). Figure 2A As described above). Additionally, as shown in user interface 502 (and according to...), Figure 5 In the inspector scenario shown, the user accepts the prompt, which causes computing device 102-1 (e.g., via Bluetooth or Wi-Fi) to transmit a message to peripheral computing device 102-2. This information may indicate acceptance of pairing and / or include information indicating completion of pairing or setup.

[0063] like Figure 5 As shown, another exemplary user interface 504 can be presented based on information in the light signal. User interface 504 can present a "Low Battery!" warning. This warning can be included as a code along with the protocol defining the code in the information of the light signal. Computing device 102-1 can use the protocol to determine what the code means and present the information on user interface 504. Furthermore, a graphical icon is presented on user interface 504, which allows the user to select an option to receive further information related to the warning. In this example, the information may describe how the user can properly charge peripheral computing device 102-2.

[0064] like Figure 5 As shown, another exemplary user interface 506 can be presented based on information in the optical signal. User interface 506 can present an error or status such as "Low Battery!". This error or status can be included as a code along with the protocol defining the code in the information of the optical signal. Computing device 102-1 can use the protocol to determine what the code means and present the information on user interface 506. Furthermore, a graphical icon is presented on user interface 506, which allows the user to select an option to perform an action related to the error or status. In this example, selecting an option allows computing device 102-1 to begin diagnosing the cause of the Wi-Fi disconnection.

[0065] therefore, Figure 5 The embodiments described herein illustrate a way to determine a user’s intention to engage with his or her peripheral computing device 102-2 when computing device 102-1 is within physical proximity of peripheral computing device 102-2 that is transmitting optical signals encoded in a high frequency.

[0066] Furthermore, it should be noted that the embodiments described herein can be modified to achieve the same or similar results using different methods without departing from the scope of this disclosure. For example, audio signals or other suitable signals can be used to transmit information to computing device 102-1. As another example, computing device 102-1 can be configured to ignore any audio signals generated by peripheral computing device 102-2 when receiving optical signals. Larger sampling rates, symbol rates, bit rates, and block coding sizes can be employed.

[0067] Optical signals can effectively convey information using any analog encoding scheme. The computing device 102-1 then receives the optical signal and extracts information from it according to the encoding scheme used. According to some implementations, an initial handshake can be performed between the peripheral computing device 102-2 and the computing device 102-1 to convey an instruction on the encoding scheme to be used. In this respect, the encoding scheme can be dynamically changed to help increase security and prevent malicious activity.

[0068] Additionally, it should be noted that the optical signals described herein can encompass any form of visual pattern animation displayed on a display device. For example, peripheral computing device 102-2 may include a display device (e.g., an integrated liquid crystal display (LCD) screen, an organic light-emitting diode (OLED) screen, etc.) or communicate with an external display device (e.g., a television set) capable of displaying animation to which information is encoded. For example, when an audio signal (and encoded information) is used, the animation may be consistent with timing information included in the audio signal. Similarly, when no audio signal is used, or when these audio signals do not play a holistic role in the pairing process, the animation itself may encode information. In any case, computing device 102-1 may be configured to acquire and process animation (alone or in combination with audio signals) to effectively perform the techniques described herein. Again, it should be noted that the above examples do not represent an exhaustive list of different methods that can be used. Furthermore, it should be noted that the techniques associated with these examples can be combined / modified in any way without departing from the scope of this disclosure.

[0069] Additionally, it should be noted that the techniques described herein may include out-of-band verification of the authenticity of peripheral computing device 102-2. This may involve, for example, computing device 102-1 facilitating communication between peripheral computing device 102-2 and a server device (e.g., managed by the manufacturer of peripheral computing device 102-2 or a partner of that manufacturer) to enable the server device / computing device 102-1 to verify the authenticity of peripheral computing device 102-2. For example, the server device may issue cryptographic challenges to peripheral computing device 102-2, which may only be correctly responded to by peripheral computing device 102-2. Furthermore, peripheral computing device 102-2 may provide identifier information, encryption key information, etc., as an indication of its authenticity. The server device can then instruct computing device 102-1 whether a pairing process should be performed, thereby significantly enhancing security. Furthermore, it should be noted that peripheral computing device 102-2 may employ similar techniques to verify the authenticity of computing device 102-1 to reduce the likelihood of participating in malicious pairing. For example, peripheral computing device 102-2 can issue a challenge to computing device 102-1 (e.g., as described above) to verify authenticity. Furthermore, if an internet connection is available to peripheral computing device 102-2, peripheral computing device 102-2 can communicate with a server device to perform an additional level of verification for computing device 102-1.

[0070] Figure 6 Detailed views of a computing device 600 according to some embodiments are shown, which may represent various techniques for implementing the techniques described herein. Figure 1 Computing devices. For example, this detailed view shows a combination of... Figure 1 The computing device 102 may include various components. For example... Figure 6As shown, computing device 600 may include a processor 602 representing a microprocessor or controller for controlling the overall operation of computing device 600. Computing device 600 may also include a user input device 608 that allows a user of computing device 600 to interact with computing device 600. For example, user input device 608 may take various forms, such as buttons, keypads, dial pads, touchscreens, audio input interfaces, visual / image capture input interfaces, sensor data input, etc. Additionally, computing device 600 may include a display 610, which may be controlled by processor 602 (e.g., via a graphics component) to display information to the user. Data bus 616 facilitates data transfer between at least storage device 640, processor 602, and controller 613. Controller 613 can be used to interface with and control different devices via equipment control bus 614. Computing device 600 may also include a network / bus interface 611 coupled to data link 612. In the case of wireless connectivity, network / bus interface 611 may include a wireless transceiver.

[0071] As described above, computing device 600 also includes storage device 640, which may include a single disk (e.g., a hard disk) or a collection of disks. In some embodiments, storage device 640 may include flash memory, semiconductor (solid-state) memory, etc. Computing device 600 may also include random access memory (RAM) 620 and read-only memory (ROM) 622. ROM 622 may store executable programs, utilities, or processes in a non-volatile manner. RAM 620 may provide volatile data storage and store instructions related to the operation of applications executing on computing device 600.

[0072] Aspects, embodiments, specific implementations, or features of the described embodiments may be used individually or in any combination. Aspects of the described embodiments may be implemented by software, hardware, or a combination of hardware and software. The embodiments may also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of such computer-readable media include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, hard disk drive, solid-state drive, and optical data storage devices. Computer-readable media may also be distributed across network-coupled computer systems, such that the computer-readable code is stored and executed in a distributed manner.

[0073] For illustrative purposes, the foregoing description uses specific names to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that specific details are not required to practice the described embodiments. Therefore, the foregoing description of specific embodiments is presented for illustrative and descriptive purposes. The foregoing description is not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to those skilled in the art that many modifications and variations are possible in light of the teachings above.

Claims

1. A method for enabling a computing device to securely communicate with a peripheral computing device, the method comprising, at the computing device: Receive a notification from a peripheral computing device, wherein the notification indicates that an intentional gesture has been made; Receives an optical signal from the peripheral computing device, wherein the optical signal is received at a detection sensor of the computing device at a first frequency higher than a second frequency detectable by the camera system of the computing device; and In response to determining that the amount of time between receiving the notification and receiving the optical signal meets a threshold: Extracting information from the optical signal; and Use the information to perform the operation.

2. The method of claim 1, wherein satisfying the threshold indicates an intention gesture by which a user of the computing device engages with the peripheral computing device.

3. The method of claim 1, further comprising, before performing the operation: Display the option to approve the operation, and Receive the selection of the option.

4. The method of claim 3, further comprising responding to receiving the selection of the option: Indicate approval for the operation to the peripheral computing device.

5. The method according to claim 1, wherein: The information is encoded in digital packets of the optical signal using analog modulation and coding techniques, including time-domain modulation, frequency-shift keying, or some combination thereof, and The digital grouping includes segments for error correction.

6. The method of claim 1, wherein the operation comprises: Pair the computing device with the peripheral computing device. Identify quick-response QR codes, Identify the status of the peripheral computing device. Detection error or warning, or Some combination of them.

7. The method of claim 1, wherein the optical signal is generated by at least one light source included in the peripheral computing device.

8. The method of claim 1, wherein the optical signal is modulated using a frequency of at least 3200 Hz.

9. The method according to claim 1, further comprising: The detection sensor is used to prevent ambient light from interfering with the data captured by the camera system.

10. The method of claim 1, wherein the first frequency is in the range of 50 Hz to 4 kHz.

11. The method of claim 1, further comprising receiving a second optical signal at the second frequency simultaneously with the optical signal, wherein the second frequency is perceptible to humans.

12. The method according to claim 1, further comprising: The system receives the selection of variable parameters, which configure the number of times the optical signal is retransmitted by the peripheral computing device.

13. The method of claim 1, wherein the optical signal is received from the peripheral computing device as an asymmetric, unidirectional communication.

14. The method of claim 1, wherein the information includes an identifier of the protocol and code defined by the protocol, and the method further comprises: The operation to be performed is determined based on the protocol and the code.

15. A method for enabling a peripheral computing device to securely communicate with a computing device, the method comprising, at the peripheral computing device: Transmit a notification that can be received by the computing device, wherein the notification indicates that an intentional gesture has been made; Encoding information into optical signals; and The optical signal is transmitted to the computing device, wherein the optical signal: The signal is received at a first frequency higher than a second frequency detectable by the camera system of the computing device at the detection sensor of the computing device; and The computing device responds to determining that the amount of time between receiving the notification and receiving the optical signal satisfies a threshold: Extracting the information from the optical signal, and Use the information to perform the operation.

16. The method of claim 15, wherein the information includes: A request to pair the peripheral computing device with the computing device. The model of the peripheral computing device. The protocol used by the peripheral computing device, Quick response QR code, The status of the peripheral computing device. Error or warning, or Some combination of them.

17. A computer-readable storage medium storing instructions that, when executed by a processor included in a computing device configured to securely communicate with a peripheral computing device, cause the computing device to perform the method according to any one of claims 1 to 14.

18. A computer-readable storage medium storing instructions that, when executed by a processor included in a peripheral computing device configured to securely communicate with a computing device, cause the peripheral computing device to perform the method according to any one of claims 15 to 16.

19. A computing device configured to securely communicate with peripheral computing devices, the computing device including at least one processor configured to cause the computing device to perform the method according to any one of claims 1 to 14.

20. A peripheral computing device configured to securely communicate with a computing device, the peripheral computing device including at least one processor configured to cause the peripheral computing device to perform the method according to any one of claims 15 to 16.

Images