Broadcasting Contextual Information by Modulating Audio and Visual Interfaces

Abstract

Presented here is an approach to transfer relevant information from a first electronic device (also called a "source electronic device") currently outputting an interface signal detectable by a second electronic device (also called a "recipient electronic device") via modulation of the interface signal. By such modulation, the relevant information is added to the interface signal, which is then forwarded to the recipient electronic device. This can be achieved by superimposing another signal representing the relevant information onto the interface signal. Thus, by superimposing a signal onto the interface signal output by the source electronic device, the relevant information can be transferred from the source electronic device to the recipient electronic device.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 167,461, entitled “Contextual Data Broadcasts Via Audio-Visual Interface Modifications,” filed March 29, 2021, and incorporated herein by reference in its entirety.

[0002] (Technical field) This disclosure relates to wireless communication of information, and more particularly, to approaches for wirelessly communicating information between electronic devices through modulation of audibly or visually detectable content. [Background technology]

[0003] With the development of the Internet of Things (IoT), electronic devices become constantly available and individuals expect to be constantly connected to the Internet, their electronic devices, and each other. Historically, this has been accomplished through wireless networking protocols collectively known as "Wi-Fi." These wireless networking protocols allow multiple nearby electronic devices to exchange data using radio waves.

[0004] The demand for greater connectivity has led to an increased expectation that nearly every electronic device will have the capability to process audio or video in real time and perform digital signal processing locally. With such capabilities in mind, companies have begun to develop new technologies that facilitate wireless interaction between electronic devices. One of these technologies is data-over-sound. Data-over-sound enables the exchange of data between a first electronic device that contains an audio output component (also known as a "source electronic device") and a second electronic device that contains an audio input mechanism (also known as a "recipient electronic device").

[0005] At a high level, Data-over-Sound works similarly to a Quick Response (QR) code, except that the data is transferred over an acoustic channel. In practice, data is encoded into an acoustic signal by a first electronic device, for example as a series of tones, to form an "acoustic barcode." The first electronic device then emits the acoustic signal for reception by a second electronic device. Upon receipt, the second electronic device demodulates the acoustic signal to decode the data encoded therein. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 contains a high level diagram showing how relevant information may be overlaid onto an interface signal output by a source electronic device.

[0007] [Diagram 2] FIG. 2 includes a high-level representation of a system that can be used to broadcast machine-interpretable encoded data from an electronic device to one or more other electronic devices.

[0008] [Diagram 3] FIG. 3 illustrates an example of an electronic device that can implement a manipulation platform designed to manipulate interface signals to transfer information to a recipient electronic device.

[0009] [Figure 4] FIG. 4 includes a schematic diagram of a procedure for transferring information from a first electronic device (also called a “source electronic device”) to a second electronic device (also called a “recipient electronic device”) through modulation of an audible or visual signal.

[0010] [Diagram 5] FIG. 5 includes a high level diagram of the communication between a source electronic device and a recipient electronic device.

[0011] [Figure 6] FIG. 6 contains a flow diagram of a process performed by a source electronic device to transfer information to a recipient electronic device.

[0012] [Figure 7A] FIG. 7A includes a schematic diagram illustrating how an extraction algorithm can process an interface signal to identify an appropriate opportunity to transfer information to a recipient electronic device.

[0013] [Figure 7B] FIG. 7B includes a schematic diagram illustrating how a detection algorithm can examine the output generated by an extraction algorithm to determine whether it is appropriate to initiate a transfer of information from a source electronic device to a recipient electronic device.

[0014] [Figure 7C] FIG. 7C includes a schematic diagram illustrating how a modulation algorithm can modulate an interface signal to overlay relevant information thereon.

[0015] [Figure 7D] FIG. 7D includes a schematic diagram illustrating how a superposition algorithm can superpose a second signal output by a modulation algorithm onto a first signal broadcast by a source electronic device.

[0016] [Figure 8] FIG. 8 is a diagram illustrating how links can be intelligently embedded into an audio signal based on an analysis of the sounds contained in the audio signal.

[0017] [Figure 9] FIG. 9 illustrates how the approaches described herein can be used to facilitate secure transfer of sensitive information.

[0018] [Figure 10]FIG. 10 is a block diagram illustrating an example of a processing system capable of implementing at least some of the operations described herein.

[0019] Various features of the technology described herein will become more apparent to those skilled in the art from a consideration of the detailed description in conjunction with the drawings. Embodiments are shown in the drawings by way of example, and not by way of limitation. While the drawings depict various embodiments for illustrative purposes, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Thus, while specific embodiments are shown in the drawings, the technology is susceptible to various modifications. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] There are several advantages to transferring data by sound, a concept commonly referred to as "acoustic data transmission" or "data over sound." Although acoustic data transmission can be accomplished using audible sound signals, acoustic data transmission is typically accomplished using inaudible sound signals (also called "ultrasonic signals"). As such, acoustic data transmission may be largely, if not completely, sensorily imperceptible to nearby individuals. Furthermore, since acoustic connectivity relies on audio output and input components, many electronic devices in use today can participate in data over sound. Simply put, acoustic data transmission is compatible with electronic devices of various form factors, architectures, etc. As an example, an ultrasonic signal can be broadcast by a given electronic device as an "advertising cookie," which can be collected and examined by an "always-on" audio input component of a nearby electronic device. Acoustic data transmission is also secure, since unauthorized access requires a responsible party (also called a "hacker") to be located in the vicinity of the electronic device from which the acoustic signal is emitted in order to access the data encoded therein.

[0021] However, acoustic data transmission has several drawbacks.

[0022] As an example, acoustic data transmission does not require a network such as the Internet, but implementation of the necessary framework generally requires a connection to a network. For example, assume that data is transferred audibly from a source electronic device to a recipient electronic device. In such a scenario, the source electronic device may require a framework that indicates, for example, when or how to transfer data to the recipient electronic device. The term "framework" may refer to a collection of algorithms, heuristics, or rules that control or influence how acoustic data transmission is performed by the source electronic device. Generally, the framework is implemented by software.

[0023] However, many electronic devices cannot initiate a connection with a network. As an example, despite the relatively rapid adoption of network-connected point-of-sale systems (also called "payment systems"), many legacy payment systems remain in place. In fact, studies indicate that approximately 93% of payment systems deployed in the United States in 2020 are not "cloud-enabled." Similarly, the majority of legacy advertising systems are not "cloud-enabled." However, these legacy payment and advertising systems typically include either an audio output component or a video output component.

[0024] As another example, acoustic data transmission may be impractical or impossible in some situations. In particular, acoustic data transmission has not historically been suitable for implementation in streaming audible content to one or more recipient electronic devices. For this reason, acoustic data transmission is difficult to implement in time-sensitive situations such as sending receipts for payment transactions and delivering advertisements, because the data transferred is related to a given point in time (and therefore timeliness is essential).

[0025] Thus, presented here is an approach to transfer relevant information from a source electronic device currently outputting an interface signal to a recipient electronic device via modulation of the interface signal. By such modulation, the relevant information is added to the interface signal and then transferred to the recipient electronic device. This can be achieved by superimposing another signal representing the relevant information onto the interface signal. Thus, by superimposing a signal onto the interface signal output by the source electronic device, the relevant information can be transferred from the source electronic device to the recipient electronic device.

[0026] As will be further described below, the nature of the interface signal (and thus the superimposed signal representing the relevant information) may depend on the capabilities of the source and recipient electronic devices. For example, assume that the source electronic device includes (i) an audio output component capable of outputting an audio signal, and (ii) a video output component capable of outputting a video signal. In such a scenario, the interface signal may represent an audio signal emitted by the audio output component, or the interface signal may represent a video signal displayed by the video output component.

[0027] Thus, the superimposed signal may be included in a manner that depends on the context and content of the interface signal, the nature of the source electronic device, or the nature of the recipient electronic device. In some embodiments, the relevant information is transferred by designing an ultrasound signal that encodes the relevant information in a manner that is audibly detectable by the recipient electronic device, but not audibly detectable by an individual associated with the recipient electronic device. In other embodiments, the relevant information is transferred by designing a video modification that encodes the relevant information in a manner that is visually or otherwise detectable by the recipient electronic device, but not visually detectable by the corresponding individual. For example, the video modification may be designed and implemented to be detectable by a radar system included in the recipient electronic device. One example of a radar system is a millimeter wave (mm wave) radar system, which includes (i) a transmitter capable of generating millimeter (mm) range electromagnetic waves, (ii) a first antenna capable of transmitting mm range electromagnetic waves, and (iii) a second antenna capable of receiving mm range electromagnetic waves. In general, the terms "mm range" and "mm wave range" refer to portions of the electromagnetic spectrum between 24 gigahertz (GHz) and 300 GHz. A second detection method of interest is Optical Camera Communication (OCC), where the video modulation is designed according to one of the schemes known in the art so as to be detectable to an image sensor included in an electronic device that has a visual line of sight to the display on which the video modulation is presented.

[0028] In some embodiments, the content of the overlay signal is determined based on a real-time analysis of the content of the interface signal. For example, the overlay signal may include information about a product mentioned or shown in the interface signal, or the overlay signal may include a digital receipt of a payment transaction facilitated or completed using the interface signal. Additionally or alternatively, the timing of the overlay signal may be determined based on a real-time analysis of the content of the interface signal. Stated differently, the portion of the interface signal on which the overlay signal is overlaid may depend on the content of the interface signal. For example, if the overlay signal includes information about a product mentioned or shown in the interface signal over a time interval, the overlay signal may be combined with the interface signal over that time interval, allowing access to the information when the product is mentioned or shown. As another example, if the overlay signal includes a digital receipt of a payment transaction facilitated or completed using the interface signal, the overlay signal may be combined with the interface signal in response to a determination, based on the analysis of the interface signal, that the payment transaction has been completed.

[0029] The superimposed signal may also be uniquely linked to a recipient electronic device (and thus an individual associated with the recipient electronic device). The superimposed signal may be uniquely linked to a recipient electronic device via timing, content, and other features such as location (e.g., determined based on Global Positioning System coordinates, Internet Protocol address, etc.). Thus, (i) the content of the first signal, (ii) the content of the second signal, or (iii) the manner in which the second signal is superimposed on the first signal, uniquely associates the second signal with a recipient electronic device. For example, assume that a source electronic device is tasked with forwarding information to multiple recipient electronic devices. In such a scenario, information relevant to each recipient electronic device may be superimposed on a different portion of the interface signal. This allows the source electronic device to convey information sequentially to multiple recipient devices.

[0030] Reception of the interface signal by the recipient electronic device and subsequent decoding of the overlaid signal can also enable other functionality. For example, certain user interface (UI) features can be made accessible for interaction on the recipient electronic device for a certain period of time (e.g., as long as the overlaid signal is detectable by the recipient electronic device). In this manner, the overlaid signal can act as a beacon that allows for providing contextualized data based on location.

[0031] For purposes of explanation, embodiments may be described in the context of transferring relevant information in a given context. For example, embodiments may be described in the context of supplementing advertisements with relevant information or providing digital receipts for payment transactions. However, the approaches described herein may be applied to other contexts that benefit from the wireless transfer of data from one electronic device to another electronic device as well.

[0032] Although not required, implementations are described below in the context of instructions executable by an electronic device. The term "electronic device" is generally used interchangeably with the term "computing device," and thus may be used to refer to computer servers, payment systems, advertising systems, tablet computers, mobile phones, wearable devices (e.g., fitness trackers and watches), and the like.

[0033] Although aspects of the technology, such as particular modules, may be described as being performed exclusively or primarily by a single electronic device, some implementations are performed in a distributed environment in which modules are shared among multiple electronic devices linked through a network. For example, an individual may indicate that they need additional information about a product displayed by the advertising system. In such a scenario, the advertising system may retrieve the information from a computer server accessible over a network (e.g., the Internet) and then forward the information to a mobile phone associated with the individual as described above. Alternatively, the information may be stored on the advertising system, so that the advertising system need not be connected to a network. [term]

[0034] References herein to "one embodiment" or "one embodiment" mean that a described feature, function, structure, or characteristic is included in at least one embodiment of the technology. Appearances of such phrases do not necessarily refer to the same embodiment, nor do they necessarily refer to alternative embodiments that are mutually exclusive.

[0035] Unless the context clearly requires otherwise, the terms "comprises," "comprising," and "consisting of" are intended to be interpreted in an inclusive sense (i.e., in the sense of "including but not limited to") and not in an exclusive or exhaustive sense. The term "based on" is also intended to be interpreted in an inclusive sense and not in an exclusive or exhaustive sense. Thus, unless otherwise specified, the term "based on" is intended to mean "based at least in part on."

[0036] The terms "connected," "coupled," and variations thereof are intended to include any connection or coupling between two or more elements, either directly or indirectly. The connection / coupling may be physical, logical, or a combination thereof. For example, objects may be electrically or communicatively coupled to each other even though they do not share a physical connection.

[0037] The term "module" may refer to a software component, a firmware component, or a hardware component. A module is typically a functional component that produces one or more outputs based on one or more inputs. As an example, a computer program may contain multiple modules responsible for completing different tasks, or a single module responsible for completing all tasks.

[0038] When used in connection with a list of multiple items, the term "or" is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0039] The order in which steps are performed in any of the processes described herein is exemplary. However, steps may be performed in various orders and combinations, unless this is contrary to physical feasibility. For example, steps may be added to or deleted from the processes described herein. Similarly, steps may be interchanged or reordered. As such, the description of any process is intended to be open-ended. [Introduction to information transmission through signal manipulation]

[0040] 1 includes a high-level diagram illustrating how relevant information may be overlaid on an interface signal output by a source electronic device 102. As mentioned above, the interface signal may be an audio signal 104 output by an audio output component 106, or the interface signal may be a video signal 108 output by a video output component 110.

[0041] At a high level, the approach introduced here allows relevant information to be superimposed on the interface signal to be transferred to the recipient electronic device 112. The source electronic device 102 and the recipient electronic device 112 may have a network connection (e.g., via respective antennas), but these devices may not be able to connect directly to each other. Additionally or alternatively, the source electronic device 102 may not be able to determine which electronic devices, if any, are located within a given proximity range to enable a link over a wireless communication channel. The approach introduced here allows relevant information to be transferred from the source electronic device 102 to the recipient electronic device 112 even in such a scenario.

[0042] As described further below, the context detection algorithm 114 (or simply the "detection algorithm") may be responsible for detecting the context of the content of the interface signal. Based on the context, relevant information may then be identified which is broadcast to the recipient electronic devices 112. Typically, the content is analyzed by the detection algorithm 114 locally (i.e., on the source electronic device 102), although the content may also be examined elsewhere by the detection algorithm 114. For example, the detection algorithm 114 may, in some embodiments, be executed by another electronic device (e.g., a computer server) to which the source electronic device is communicatively connected.

[0043] An appropriate modulation designed to transmit the relevant information can then be identified by the superposition generation system 116. As described below with reference to FIG. 2, the superposition generation system 116 can include a modulation algorithm, a superposition algorithm, a signal database, or any combination thereof. An appropriate modulation can be designed to be superimposed on the interface signal such that the relevant information is broadcast to nearby electronic devices, such as the recipient electronic device 112. For example, the relevant information can be broadcast via a modulation of the audio signal 104 such that the modulation is detectable by an audio input component. As another example, the relevant information can be broadcast via a modulation of the video signal 108 such that the modulation is detectable by a video input component or other sensor (e.g., a millimeter wave radar system).

[0044] For ease of explanation, the modulation is shown as a binary string, i.e., 0101001101, that can be superimposed on the interface signal to generate the manipulated signal. By superimposing the modulation on the audio signal 104, the superimposition generation system 116 can generate a manipulated audio signal 118 that can be broadcast for detection by the recipient electronic device 112. Similarly, by superimposing the modulation on the video signal 108, the superimposition generation system 116 can generate a manipulated video signal 120 that can be broadcast for detection by the recipient electronic device 112.

[0045] Typically, the overlay generation system 116 manipulates only one of the audio signal 104 and the video signal 108. However, there may be scenarios in which the overlay generation system 116 manipulates the audio signal 104 and the video signal 108. For example, if the manipulation platform of which the detection algorithm 114 and the overlay generation system 116 are a part is unsure which of these signals the recipient electronic device 112 can detect, the manipulated audio signal 118 and the manipulated video signal 120 may be broadcast for detection by the recipient electronic device 112. As another example, if the manipulation platform is interested in splitting the relevant information (e.g., for privacy purposes, due to the bandwidth or delay of those channels, etc.), the manipulated audio signal 118 and the manipulated video signal 120 may be broadcast for detection by the recipient electronic device 112. In embodiments in which the manipulated audio signal 118 and the manipulated video signal 120 are broadcast for detection, the signals may contain the same or different information.

[0046] As shown in FIG. 1, the recipient electronic device 112 can detect the manipulated signal broadcast by the source electronic device 102. If the manipulated signal is a manipulated audio signal 118, the recipient electronic device 112 can detect the manipulated signal via an audio input component. If the manipulated signal is a manipulated video signal 120, the recipient electronic device 112 can detect the manipulated signal via a video input component or other sensor (e.g., a millimeter wave radar system that can sense the state of a video screen). Data can then be decoded from the manipulated signal by the recipient electronic device. Based on this data and other characteristics (e.g., timing and location), relevant information can be provided to the user of the recipient electronic device 112. Examples of relevant information include personalized offers for brands, products, and merchants, digital receipts for payment transactions, etc. [Outline of the signal operation platform]

[0047] FIG. 2 includes a high-level representation of a system 200 that can be used to broadcast machine-interpretable encoded data from an electronic device 204 to one or more other electronic devices (not shown). For purposes of explanation, the electronic device 204 may be referred to as a “source electronic device” and the other electronic devices may be referred to as “recipient electronic devices.” To broadcast the machine-interpretable encoded data, the signal manipulation platform 202 (or simply “manipulation platform”) can manipulate an interface signal output by the electronic device 204. For example, the manipulation platform 204 may manipulate a video signal displayed by a video output component 206 of the electronic device 204, or the manipulation platform 204 may manipulate an audio signal emitted by an audio output component 208 of the electronic device 204. These manipulations may be designed and implemented such that the resulting changes in the video or audio signals are largely, if not completely, sensory imperceptible to individuals associated with the recipient electronic device. These individuals may be referred to as “users” of the recipient electronic device.

[0048] It should be noted that the term "sensory perceptible" may refer to stimuli that can be seen, noticed, or detected by typical human senses. Stimuli may be visually perceptible, auditorily perceptible, tactilely perceptible, etc. Examples of visual stimuli include still and moving images, and examples of audible stimuli include audiobooks, podcasts, and radio broadcasts. Some stimuli may be perceptible by more than one sense (e.g., a video signal may be visually and auditorily perceptible). Thus, the term "sensory non-perceptible" may be used to refer to stimuli that cannot be seen, noticed, or detected by typical human senses.

[0049] For example, assume that an individual visits a retail establishment where audible content is being emitted. As the individual walks through the retail establishment, the underlying audio signal may be manipulated to include information about products offered by the retail establishment or the retail establishment itself. For example, the information may be a hyperlink (or simply a "link") to a website containing details about products or promotional campaigns offered by the retail establishment. This results in the underlying audio signal being manipulated, but the manipulation may not be detectable by the human ear. Thus, although the electronic device may detect the manipulation, the individual may not be aware that the underlying audio signal has been manipulated. By decoding the manipulated audio signal, the electronic device can "discover" the information. A similar process for transferring information via modulation of audible content may also be performed when an individual is listening to a podcast at home, listening to the radio in their car, walking through a public environment such as a shopping center or airport, etc.

[0050] As another example, assume that an individual is traversing a public or private environment while wearing an augmented reality headset (also referred to as a "mixed reality headset"). As the individual traverses the environment, the individual may observe display panels that visually present content related to products, people, events, and the like. As further described below, the visual content displayed on these display panels may be manipulated to include information about those products, people, events, and the like. For example, if the content is about a musical performance by an artist, the information may be a link to a website where tickets to the musical performance can be purchased. As another example, if the content is about a movie showing in a theater, the information may be a link to a trailer for the movie. Again, the manipulation causes a change in the underlying video signal, but the manipulation may not be detectable by the human eye. Thus, while the mixed reality headset may detect the manipulation, the individual may not be aware that the underlying video signal has been manipulated. By decoding the manipulated video signal, the mixed reality headset can "discover" the information. Comparable processing may be performed by other types of electronic devices that include sensors (e.g., image sensors) that can visually monitor the environment. Examples of such electronic devices include mobile phones, wearable cameras, and some watches and fitness accessories, etc. As a specific example, a mobile phone may be able to detect modulation of visual content when panned about an environment that includes a display panel.

[0051] As shown in FIG. 2, the system 200 includes an operating platform 202 that may access a video output component 206, an audio output component 208, a processor 210, or any combination thereof. One skilled in the art will recognize that not all of these components necessarily need to be included in the electronic device 204. For example, the electronic device 204 may include an audio output component 208 in some embodiments but not a video output component 206. In other embodiments, the electronic device 204 may include a video output component 206 but not an audio output component 208. The electronic device 204 may include other components in addition to those shown in FIG. 2. For example, the electronic device 204 may include an audio input component in addition to the audio output component 208 to enable audio interaction between the electronic device 204 and recipient electronic devices or users of those devices. One example of an audio input component is a microphone designed to convert sound into electrical impulses to create signals that can be processed by other components of the electronic device 204.

[0052] The video output component 206 may be any mechanism operable to visually convey the contents of a video signal to an individual and a corresponding recipient electronic device. For example, the video output component 206 may be a display panel including light emitting diodes (LEDs), organic LEDs, liquid crystal elements, or electrophoretic elements.

[0053] The audio output component 208 may be any mechanism operable to audibly convey the contents of the audio signal to an individual and a corresponding recipient electronic device. For example, the audio output component 208 may be a loudspeaker (or simply a "speaker") designed to convert electrical impulses into sound.

[0054] The electronic device 204 may be referred to as an "audio-video device" or an "audio-visual device" because it can output audio signals at the audio output component 208 and video signals at the video output component 206. As described above, in operation, the electronic device 204 may output interface signals detectable by a user. In embodiments in which the interface signals represent audio signals, the electronic device 204 may be described as generating or supporting an "audio interface" with which a user can interact. In embodiments in which the interface signals represent video signals, the electronic device 204 may be described as generating or supporting a "video interface" or a "visual interface" with which a user can interact. In some scenarios, the electronic device 204 may output multiple interface signals, e.g., an audio signal output by the audio output component 208 and a video signal output by the video output component 206. In such embodiments, the electronic device 204 may be described as generating or supporting an "audio-video interface" or an "audio-visual interface" with which a user can interact. As described further below, the operating platform 202 may operate on one or both of these interface signals.

[0055] As shown in FIG. 2, the operation platform 202 may include a signal modulation algorithm 212 (or simply a “modulation algorithm”), a signal superposition algorithm 214 (or simply a “superposition algorithm”), and a signal database 216. The signal database 216 may store various signals that represent information that can be shared with a recipient electronic device. In other words, the signal database 216 may store various signals that can be superimposed on an interface signal output by the electronic device 204. As will be further described below, the signals in the signal database 216 may vary depending on the nature of the interface signal output by the electronic device 204, the components of the electronic device 204, or the components of the recipient electronic device. Thus, the signal database 216 may include audio signals, video signals, or a combination thereof. The signals in the signal database 216 may be encrypted, hashed, or obfuscated to prevent unauthorized access.

[0056] In some embodiments, the manipulation platform 202 is configured to generate the signal superimposed on the interface signal in near real time. For example, the manipulation platform 202 may generate the signal superimposed on the interface signal based on a real-time analysis of the content of the interface signal. As another example, the manipulation platform 202 may generate the signal superimposed on the interface signal based on one or more characteristics of the recipient electronic device (e.g., learned through analysis of communications sent from the recipient electronic device to the electronic device 204 in the form of one or more data packets). In embodiments in which the signal is generated by the manipulation platform in near real time, the signal may not be stored in the signal database 216 at all, or the signal may be stored in the signal database 216 only for a short time (e.g., a few seconds or less) until the transfer of information is completed.

[0057] When executed by the processor 210, algorithms implemented in the manipulation platform 202 enable information to be transferred from the electronic device 204 to a recipient electronic device via manipulation of audible or visually perceptible interface signals. To accomplish this, the manipulation platform 202 may complete two phases, each of which may correspond to a different algorithm, as shown in FIG.

[0058] In a first phase (also called the "modulation phase" or "creation phase"), the modulation algorithm 212 can create a signal representing the information to be shared with the recipient electronic device. This can happen in a variety of ways. In some embodiments, the modulation algorithm 212 retrieves a signal from the signal database 216 based on the content of the interface signal or the characteristics of the recipient electronic device or its user. This signal can be used without any modulation (i.e., in its "raw form"), or the signal can be modulated by the modulation algorithm 212 to be tailored to the interface signal, the recipient electronic device, or the user of the recipient electronic device. In other embodiments, the modulation algorithm 212 generates the signal in near real time based on an analysis of the interface signal on which it is superimposed.

[0059] In the second phase (also called the "superposition phase" or "transfer phase"), the superposition algorithm 214 can superpose the signal onto the interface signal such that the information to be transferred to the recipient electronic device is "carried along" with the interface signal. The superposition can be performed such that the interface signal undergoes only minimal degradation. Thus, any modulation resulting from the superposition of the signal onto the interface signal may be largely, if not completely, sensorily imperceptible to a user of the recipient electronic device, but is easily detectable by the recipient electronic device itself.

[0060] As mentioned above, in some embodiments, the interface signal may represent an audio signal. Thus, by superimposing a second audio signal (e.g., an ultrasonic audio signal) on the audio signal that serves as the interface signal, information may be transferred from the electronic device 204 to a recipient electronic device. In such an embodiment, the timing of the information transfer may be optimized based on an analysis of the content (e.g., spoken words, non-lexical speech, etc.) contained in the interface signal. Stated differently, the portion of the interface signal on which the second audio signal is superimposed can be determined based on an analysis of the content of the interface signal. Additionally or alternatively, the portion of the interface signal may be specified by or determined based on an input received by the operation platform 202 indicating that information is to be transferred. Those skilled in the art will recognize that in some circumstances, an interface signal may accompany another interface signal. For example, the electronic device 204 may output a first interface signal including audio content related to a product or service and a second interface signal including visual content related to the product or service. In such a scenario, the timing of the information transfer may be optimized based on the content of the first interface signal, the content of the second interface signal, or a combination thereof.

[0061] In other embodiments, the interface signal represents a video signal. In such embodiments, the superimposition of the second audio signal onto the video signal that serves as the interface signal may result in a visual modulation of the content. This visual modulation may be visible to the human eye, or this visual modulation may be imperceptible to the human eye but detectable by the recipient electronic device. Examples of visual modulation include watermarks, flicker patterns, and changes in color scheme. In embodiments where the interface signal represents a video signal, the timing of the information transfer may be based on an analysis of the content of the interface signal. For example, in response to a determination that the interface signal output by the video output component 206 includes a particular message (e.g., "Thank you for your purchase"), content (e.g., a website confirming that a payment transaction has been completed), etc., information may be transferred from the electronic device 204 to the recipient electronic device. As described further below, various sensors included in the recipient electronic device may be used to detect, analyze, or understand the second video signal that is superimposed onto the interface signal. For example, the manipulation platform 202 may be used to design an appropriate video signal modulation to transfer information in a manner optimized for detectability via a millimeter wave radar system.

[0062] 3 shows an example of an electronic device 300 capable of implementing a manipulation platform 310 designed to manipulate an interface signal in order to transfer information to a recipient electronic device. As mentioned above, the interface signal may represent an audio or video signal that is modulated when another signal is superimposed thereon. The modulation is detectable by the recipient electronic device to which the information is transferred, but the modulation may not necessarily be detectable by its user.

[0063] In some embodiments, the manipulation platform 310 is embodied as a computer program executed by the electronic device 300. For example, the manipulation platform 310 may reside on a payment system, in which case the modulations introduced in the interface signals may represent digital receipts communicated to individuals who have completed a payment transaction using the respective recipient electronic device. As another example, the manipulation platform 310 may reside on an advertising system, in which case the modulations introduced in the interface signals may represent links to additional information about products or services displayed on the advertising system. Those skilled in the art will recognize that aspects of the manipulation platform 310 may be distributed among multiple electronic devices. As an example, the manipulation platform 310 may reside on a first electronic device (e.g., a payment system, an advertising system, etc.), while the signal database 312 may reside on a second electronic device (e.g., a computer server).

[0064] The electronic device 300 may include a processor 302 , a memory 304 , a UI output mechanism 306 , and a communication module 308 .

[0065] The processor 302 may have general purpose characteristics similar to a general purpose processor, or the processor 302 may be an application specific integrated circuit (ASIC) that provides control functions for the electronic device 300. As shown in Figure 3, the processor 302 may be directly or indirectly coupled for communication purposes to all components of the electronic device 300. In embodiments where the operating platform 310 is embodied as a computer program, the underlying instructions may be executed by the processor 302 to enable the electronic device 300 to perform the operations described herein.

[0066] The memory 304 may be comprised of any suitable type of storage medium, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or a register. In addition to storing instructions that may be executed by the processor 302, the memory 304 may also store (i) data output by the UI output mechanism 306, and (ii) data generated by the processor 302 (e.g., when executing algorithms, heuristics, or rules of the operation platform 310). It should be noted that the memory 304 is merely an abstract representation of a storage environment. The memory 304 may be comprised of an actual integrated circuit (also called a "chip") or module.

[0067] The communication module 308 may be responsible for managing communications between components of the electronic device 300. Additionally or alternatively, the communication module 308 may be responsible for managing communications with other electronic devices (e.g., recipient electronic devices 314a-n). Thus, in some embodiments, the electronic device 300 can communicate with the recipient electronic devices 314a-n over a network, while in other embodiments, the electronic device 300 cannot communicate with the recipient electronic devices 314a-n over a network. Instead, the electronic device 300 may communicate with the recipient electronic devices 314a-n over an interface signal, as described above. Examples of the recipient electronic devices 314a-n include mobile phones, tablet computers, personal computers, wearable devices, and the like. Meanwhile, the electronic device 300 may be a mobile phone, tablet computer, personal computer, wearable device, payment system, advertising system, and the like. As an example, the electronic device 300 may be a payment system, while the recipient electronic devices 314a-n may be mobile phones associated with respective users who complete payment transactions using the payment system.

[0068] In embodiments where the communication module 308 can facilitate communication with a destination external to the electronic device 300 (e.g., recipient electronic devices 314a-n), the communication module 308 can be wireless communication circuitry designed to establish a communication channel with the destination. Examples of wireless communication circuitry include chips configured for Bluetooth, Wi-Fi, Near Field Communication (NFC), etc.

[0069] For convenience, the manipulation platform 310 may be referred to as a computer program residing in the memory 304. However, the manipulation platform 310 may be comprised of software, firmware, or hardware implemented in or accessible to the electronic device 300. In accordance with embodiments described herein, the manipulation platform 310 may include various algorithms as described above with reference to FIGS. 1-2. Typically, these algorithms are executed by separate modules of the manipulation platform 310 that are separately addressable (and thus capable of executing independently without interfering with other modules). These modules may be integral parts of the manipulation platform 310. Alternatively, these modules may be logically separate from the manipulation platform 310, but may operate "in parallel" with it. Together, these algorithms may enable the manipulation platform 310 to modulate interface signals to transfer information to a destination.

[0070] For example, assume that a person completes a payment transaction on an electronic device 300 that functions as a payment system. Traditionally, when a payment transaction is completed on a sophisticated payment system, a digital receipt is provided in the form of a text message, an email message, or a push notification, and the underlying information is transmitted from these sophisticated payment systems over a network. However, some payment systems cannot connect to a network such as the Internet. The approach described herein enables these payment systems to transfer information (e.g., a digital receipt) to a nearby recipient electronic device through modulation of an audible or visual signal. In addition to enabling a non-smart payment system to transfer information quickly with little computational resource consumption, the approach is also secure. Thus, the approach may be adopted by an electronic device even if the electronic device can connect to a network such as the Internet. As an example, the operation platform 310 may transfer information (e.g., a digital receipt) to a nearby recipient electronic device through modulation of an audible or visual signal instead of or in addition to transferring information over a network.

[0071] Other elements may also be included as part of the manipulation platform 310. The nature of these other elements may depend on the intended use of the electronic device 300 and the manipulation platform 310. For example, the UI module may be responsible for generating content to be output by the UI output mechanism 306 for presentation to one or more recipient electronic devices 314a-n. Generally, the content output by the UI output mechanism 306 is tailored to each user or each recipient electronic device. Thus, the UI output mechanism 306 may output a first interface signal for detection by the recipient electronic device 314a, a second interface signal for detection by the recipient electronic device 314b, etc.

[0072] The form of the content may depend on the nature of the UI output mechanism 306. For example, if the UI output mechanism 306 is a speaker, the content of the interface signal may relate to an audible confirmation of an event (e.g., completion of a payment transaction), an audible description of an item (e.g., a good or service), etc. As another example, if the UI output mechanism 306 is a display panel, the content of the interface signal may relate to a visual confirmation of an event (e.g., completion of a payment transaction), a visual description of an item (e.g., a good or service), etc. As described further below, the UI output mechanism 306 may also be responsible for outputting (e.g., emitting or displaying) an interface signal with another signal superimposed thereon to audibly or visually modulate the interface signal.

[0073] In operation, the manipulation platform 310, in combination with other components of the electronic device 300, may represent a system that can analyze the audible or visual content of the interface signal and embed relevant data packets (e.g., links or offers) at moments relevant to the underlying content of the interface signal. In some embodiments, the manipulation platform 310 operates in near real-time to enable use with live media content. Examples of live media content include radio shows, live television shows, live streaming events, and the like.

[0074] As an example, an extraction algorithm may be designed or trained to recognize potential speech features in the audio, such as references to brands, products, or attributes of the brand or product (also called "sound tags"), and assign a label to the audio to indicate the sound tag. Thus, different portions of the audio content may be "tagged" or "labeled" to indicate that content. Once the detection algorithm detects relevant sound tags, information about each sound tag may be forwarded to one or more nearby recipient electronic devices. For example, if the detection algorithm finds a sound tag associated with a given product, the manipulation platform 310 (more specifically, its modulation and superposition algorithms) may send a link to a website related to the given product to one or more recipient electronic devices.

[0075] Alternatively, the audible or visual content of the interface signal may be "pre-analyzed" in a static mode. In this scenario, the operation platform 310 may survey the interface signal in a similar manner, but this survey is not performed at run-time (e.g., while the interface signal is output for detection by one or more recipient electronic devices 314a-n). One advantage of "pre-analyzing" the interface signal is that less computational resources are required for labeling with sound tags, since timeliness is less of a concern. Another advantage of "pre-analyzing" the interface signal is that more robust detection algorithms can be used that cannot be implemented in near real-time without delay. Similarly, "pre-analyzing" the interface signal may enable the operation platform 202 to perform a multi-stage analysis (e.g., with a detection algorithm that identifies sound tags quickly but with relatively low accuracy and another detection algorithm that identifies sound tags slowly but with relatively high accuracy).

[0076] As will be further described below, the manipulation platform 310 may be programmed to design and then output visual modulations in addition to or instead of audible modulations. For example, the manipulation platform 310 may be able to design and then output visual modulations optimized for detection by a millimeter wave radar system. Visual modulations can be designed such that they induce large state changes in a video output component (e.g., a display panel) without being noticeable to the human eye. One example of a suitable visual modulation is a flicker pattern, where adjacent rows or columns of a display panel cycle between different color states, and information is encoded via a suitable frequency modulation scheme. A visual modulation does not necessarily have to be visually detectable (e.g., by a millimeter wave radar system), but may be easily detectable by a recipient electronic device, while visibility by a corresponding user may be minimized by selecting a carrier frequency that exceeds the flicker fusion frequency for human vision. Alternatively, a visual modulation may be noticeable to a user, but only affect a portion of an interface signal. For example, a visual modulation may be overlaid on a less important portion of a video signal, such as a top or bottom corner. Thus, the modulation may only affect the portion of each frame that is contained in the portion of the interface signal on which the signal is superimposed.

[0077] As part of the information transfer process, the manipulation platform 310 may identify one or more characteristics of the electronic device 300. Examples of characteristics include display size, display type (e.g., LCD, OLED, etc.), model, desired detection distance from the recipient electronic device, intended use of the approach described herein, nearby noise sources (e.g., radio frequency noise), etc. Based on these characteristics, the manipulation platform 310 may identify a modulation scheme. The modulation scheme may govern how information is transferred to the recipient electronic device. The modulation scheme may specify the nature of the signal superimposed on the interface signal. For example, the modulation scheme may specify characteristics of the signal, such as whether the signal is an audio signal or a video signal, whether the signal is ultrasonic, whether the video signal is intended to be detectable by a particular sensor (e.g., a millimeter wave radar system), etc. Each information transfer may be performed according to a modulation scheme, but the modulation scheme does not necessarily impose the same constraints (e.g., regarding packet size) on each recipient electronic device.

[0078] As described above, the electronic device 300 may be tasked with conveying information to multiple recipient electronic devices 314a-n simultaneously or sequentially. Since the information transfer involves an audible or visual signal, the same information may be transferred to each recipient electronic device that simultaneously "hears" or "sees" the interface signal. To transfer different information to each recipient electronic device, the recipient electronic device 314a-n may disambiguate the transmissions through timing, content, and other characteristics. For example, assume that the electronic device 300 serves as a payment system where different individuals complete payment transactions. Digital receipts for these payment transactions may be broadcast indiscriminately by the electronic device 300. In such a scenario, the recipient electronic device 314a-n may "hear" many of these transmissions. For each transmission "heard," each recipient electronic device may search a computer program (e.g., a payment application) running on that recipient electronic device to find a transaction whose information matches one of the transmissions. This allows the recipient electronic device to associate the received data (e.g., purchased items and associated costs) with the corresponding user and provide a digital receipt. The remaining transmissions may be ignored or deleted if received as irrelevant. In some embodiments, the transmissions may be differentiated based on timing. For example, the recipient electronic device may "look" for transmissions for a certain time interval after determining that a transaction is complete based on information available from a computer program (e.g., a payment application).

[0079] Reception and subsequent decoding of the interface signal by the recipient electronic device may enable various functionality. For example, for a certain period of time (e.g., as long as the overlaid signal is detectable in the interface signal), a particular UI function may be made accessible for interaction on the recipient electronic device. In this manner, the overlaid signal may function as a beacon that allows for providing contextualized data based on location. As another example, a particular UI function may be enabled on the recipient electronic device based on its movement during the period of information transfer. For example, the UI may take the form of a physical movement on the recipient electronic device relative to the electronic device 300 such that an interface signal on which another signal is overlaid is altered in its directionality and intensity. A particular action may be associated with the UI function, such as accepting or rejecting a proposed course of action (e.g., transferring information, completing a payment transaction, etc.). [Signal operation methodology]

[0080] 4 includes a schematic diagram of a procedure 400 for transferring information from a first electronic device (also called a "source electronic device") to a second electronic device (also called a "recipient electronic device") through modulation of an audible or visual signal. This audible or visual signal may be referred to as an "interface signal" or a "first signal" that is detectable by a user of the source electronic device. As mentioned above, a user may be associated with the recipient electronic device.

[0081] In some embodiments, the first signal is detectable by the recipient electronic device. For example, if the first signal includes human-audible audio content, the recipient electronic device may be able to detect the first signal using an audio input component. However, the first signal does not necessarily have to be detectable by the recipient electronic device. For example, assume that a payment system transfers data including information related to a payment transaction to a customer's mobile phone. In such a scenario, the payment system represents the source electronic device, while the mobile phone represents the recipient electronic device. The first signal may be human-readable video content displayed by the payment system. To transfer data to the mobile phone (e.g., specifying the items to be purchased and the associated costs), a signal manipulation platform running on the payment system may modulate the first signal such that the modulation is detectable by a sensor included in the mobile phone, even though the mobile phone may not currently be visually observing the payment system. For example, the modulation of the human-readable video content may be detectable by a millimeter wave radar system included in the mobile phone. This example also illustrates that the user of the source electronic device (e.g., a cashier) need not be the same person as the user of the recipient electronic device (e.g., a customer).

[0082] Initially, the manipulation platform can acquire a first signal. In some embodiments, the first signal is acquired in near real-time (e.g., in conjunction with output by the UI output mechanism 412). In other embodiments, the first signal is "pre-scanned" before being output by the UI output mechanism 412, as described above. The manipulation platform can then provide the first signal as input to the extraction algorithm 402 to obtain a set of tags. If the first signal represents an audio signal, these tags may represent "sound tags". If the first signal represents a video signal, these tags may represent "video tags". Regardless of their nature, these tags may represent labels indicative of the content of the first signal.

[0083] The operation platform can then provide the first signal and associated tags to a detection algorithm 404. Once relevant tags are detected by the detection algorithm 404, information about those tags can be forwarded to a nearby recipient electronic device. Thus, the detection algorithm 404 can analyze the tags generated by the extraction algorithm 402 to identify appropriate opportunities to forward information to the recipient electronic device. As an output, the detection algorithm 404 can generate a representation of the relevant tags. For example, the detection algorithm 404 can maintain an ordered list of tags that represent or are associated with products of interest for advertising purposes.

[0084] An indication of the associated tags may be provided as input by the operation platform to the modulation algorithm 406. As described above, the modulation algorithm 406 may be responsible for determining information to be transferred to the recipient electronic device based on the associated tags detected by the detection algorithm 404. This information may be represented as a second signal. In some embodiments, the second signal is retrieved by the modulation algorithm 406 from a signal database 408, as shown in FIG. 4. Alternatively, the modulation algorithm 406 may generate the second signal based on the tags deemed relevant by the detection algorithm 404.

[0085] The operation platform can then provide the first signal and the second signal as inputs to the superimposition algorithm 410. As described above, the superimposition algorithm 410 can superimpose the second signal onto the first signal to generate a third signal that can be output via the UI output mechanism 412 for detection by the recipient electronic device. At a high level, this third signal can represent the second signal combined with the first signal. In general, the second signal is only superimposed onto a portion of the first signal. For example, if the first signal represents an audio signal corresponding to recorded or streamed audio content, the second signal can only be superimposed onto a few seconds' worth of the audio signal. As another example, if the first signal represents a video signal corresponding to recorded or streamed video content, the second signal can only be superimposed onto a few seconds' worth of the video signal. Thus, the UI output mechanism 412 can output the third signal for a limited time (e.g., a few seconds or minutes), and the first signal can be output before and after the limited time.

[0086] 5 includes a high-level diagram of communication between a source electronic device 500 and a recipient electronic device 550. The source electronic device 500 may be the electronic device 204 of FIG. 2 or the electronic device 300 of FIG. 3. Initially, the source electronic device 500 may output a first signal (step 501) that is (i) detectable by the recipient electronic device 550 and / or (ii) detectable by a user of the source electronic device 500. As mentioned above, the user of the source electronic device 500 may be associated with the recipient electronic device 550 or the user of the source electronic device 500 may be independent of the recipient electronic device 550. In some embodiments, the first signal represents an audio signal emitted by an audio output component of the source electronic device 500. In other embodiments, the first signal represents a video signal displayed by a video output component of the source electronic device 500.

[0087] The source electronic device 500 may then receive an input indicating an instruction to communicate information to the recipient electronic device 550 (step 502). Typically, this input is generated or detected by a detection algorithm, as described above with reference to Figures 1 and 4. If the context of the first signal output by the source electronic device 500 is determined to match any of a number of pre-determined classes (also referred to as "context categories"), the detection algorithm may instruct the signal manipulation platform to initiate transmission. As an example, if the source electronic device 500 is a payment system, one context category indicating that a transfer should occur may be reaching a checkout screen. As another example, if the source electronic device 500 is an advertising system, one context category indicating that a transfer should occur may be the display of content discussing a brand, product, or merchant for which an offer is available. The input may also represent the discovery of a tag of interest, as described above. For example, the input may represent a determination that a sound tag associated with a brand, product, or merchant of interest has been detected. Alternatively, the source electronic device 500 may receive a communication from the recipient electronic device 550 that information is desired. For example, the recipient electronic device 550 may “ping” the source electronic device 500 in response to a determination (e.g., based on analysis of the first signal, input provided by a user, etc.) that information from the source electronic device 500 is desired.

[0088] The source electronic device 500 may obtain a second signal indicative of the information (step 503) and then superimpose the second signal onto the first signal to generate a third signal (step 504). As described above, the second signal is typically superimposed onto a portion of the first signal. Thus, the source electronic device 500 may output the first signal as described above with reference to step 501, output a third signal for detection by the recipient electronic device (step 505), and then output the first signal again when the second signal is no longer superimposed thereon.

[0089] How the second signal is superimposed onto the first signal may depend on the nature of the first signal, as well as the characteristics of the source electronic device 500 or the recipient electronic device 550.

[0090] In an embodiment in which the first signal represents an audio signal, the second signal may represent an ultrasonic signal that is audible to the recipient electronic device 550 but not to its user. In such a scenario, the second signal may be superimposed on the first signal by encoding the second signal into a portion of the first signal that is specified by or determined based on the input. For example, the superimposition may be performed such that the second signal (e.g., representing a digital receipt) is encoded into the first signal (e.g., representing a UI that a payment transaction has been completed) in a time-aware manner such that the information encoded in the second signal is provided to the appropriate person.

[0091] In embodiments in which the first signal represents a video signal, the superposition may include encoding the first and second signals into a form such that the encoded second signal is detectable by a sensor included in the recipient electronic device 550. The sensor may be, for example, a millimeter wave radar system designed for electromagnetic waves above 24 GHz.

[0092] As described above, the second signal may be superimposed on the first signal for a certain time interval. The operation platform may identify the time interval during which the second signal is superimposed on the first signal based on an analysis of the content of the first signal. The operation platform may then combine the first and second signals such that the second signal representing the information to be transferred coincides with the time interval.

[0093] In summary, the source electronic device 500 can output a first signal audibly or visually detectable by the recipient electronic device 550 and / or a user of the source electronic device 500, can receive input indicative of an instruction to communicate information to the recipient electronic device 550, can superimpose a second signal indicative of the information onto the first signal to generate a third signal, and can output the third signal for detection by the recipient electronic device 550.

[0094] As mentioned above, the source electronic device 500 may in some scenarios communicate with multiple recipient electronic devices. Thus, aspects of the procedure 500 may be performed simultaneously or sequentially by the source electronic device 500 to communicate with each of these recipient electronic devices. For example, steps 502-505 may be performed sequentially to transfer information to each of these recipient electronic devices. Note that the information transferred to each of these recipient electronic devices need not be the same. Thus, the second signal superimposed on the first signal may be different for each of the recipient electronic devices.

[0095] FIG. 6 includes a flow diagram of a process 600 performed by a source electronic device to transfer information to a recipient electronic device. Initially, the source electronic device can output a first stream of encoded data that is audibly or visually detectable by a user (step 601). The source electronic device can then determine that information accessible to it is to be shared with the recipient electronic device (step 602). For example, the source electronic device can determine that information should be shared with the recipient electronic device in response to determining that an event involving the user or the recipient electronic device has occurred (e.g., completion of a payment transaction, mention of a product, etc.). As a specific example, the source electronic device can implement a machine learning (ML) algorithm trained to detect the utterance of a brand name, product name, or person name included in a predefined list. The detection algorithm can generate as an output an indication that one of the brand name, product name, or person name was spoken or displayed in the first stream of encoded data. In such an embodiment, the source electronic device may determine that information about a particular brand, product, or person should be transferred to the recipient electronic device in response to a determination by the detection algorithm that a particular brand, product, or person has been spoken or displayed in the first stream of encoded data.

[0096] The source electronic device can then generate or obtain a second stream of encoded data indicative of the information to be shared (step 603). For example, assume that the information to be shared is related to a payment transaction to be completed using the source electronic device. In this scenario, the information related to the payment transaction can be extracted by the signal manipulation platform, for example, by taking numerical values ​​directly from the processor of the source electronic device, encoding them according to an appropriate encoding scheme, and then overlaying the calculated modulation onto the first stream of encoded data representing the original interface signal. More information regarding encoding schemes can be found below. Note, however, that in general, it does not matter which encoding scheme is used, as long as the recipient electronic device "knows" what to look for.

[0097] The information may then be transferred by the source electronic device to the recipient electronic device by combining the second stream of encoded data with the first stream of encoded data to create a third stream of encoded data (step 604). More specifically, the source electronic device may superimpose the second stream of encoded data onto a portion of the first stream of encoded data. This third stream of encoded data may then be output for detection by the recipient electronic device (step 605).

[0098] From the perspective of the recipient electronic device, the presence or potential presence of a broadcast may be directly detected or indirectly inferred based on its current location, its current use, or input provided by its user. For example, the recipient electronic device may activate an audio input component in response to determining that it is located within a predetermined proximity of a predetermined location. One example of an audio input component is a microphone. As another example, the recipient electronic device may activate a video input component in response to receiving input indicating a request to do so from its user. One example of a video input component is an image sensor that can generate image data, for example in the form of a digital image. The image sensor may be part of a camera module (or simply a "camera") of the recipient electronic device.

[0099] In general, the encoding and decoding schemes used by the source and recipient electronic devices, respectively, are implementation specific and therefore will not be described in detail. However, to cite one example, the IEEE (Institute of Electrical and Electronics Engineers) 802.15 series of standards defines encoding, decoding, modulation, media access, physical layer, etc. schemes for personal area networks. Any of these standards may be used to implement the mmWave and OCC embodiments described above. For embodiments that rely on acoustic data transmission, one protocol that can be used is Beeping, a communications protocol designed to transmit data between electronic devices via ultrasound. It is also possible to develop proprietary standards to implement these embodiments.

[0100] 7A-D include schematic diagrams illustrating how the extraction algorithm, detection algorithm, modulation algorithm, and superposition algorithm can work together to implement the approaches described herein. These algorithms may be equivalent to the extraction algorithm 402, detection algorithm 404, modulation algorithm 406, and superposition algorithm 410 of FIG. 4. Each of these algorithms is described in more detail below.

[0101] 7A includes a schematic diagram illustrating how an extraction algorithm 702 can process an interface signal to identify an appropriate opportunity to forward information to a recipient electronic device. The interface signal may initially be provided as an input to the extraction algorithm 702. As shown in FIG. 7A, the extraction algorithm 702 may detect the context via a number of different approaches.

[0102] In one approach (also referred to as “content-based detection”), the extraction algorithm 702 can examine the content of the interface signal (e.g., spoken audio or displayed video), extract semantic meaning from the content (e.g., meaning of numbers, letters, words, and phrases), and then determine whether the semantic meaning matches a relevant topic for data transfer. The semantic meaning can be extracted using a machine learning approach. Meanwhile, the relevant topic can be determined, for example, through a comparison of the semantic meaning to a list of brands, products, affiliated stores, etc. If the semantic meaning matches a relevant topic, the extraction algorithm 702 can output a context tag that represents the underlying content. For example, the context tag can indicate the salient numbers, letters, words, or phrases from which the semantic meaning was extracted. If the semantic meaning does not match a relevant topic, the extraction algorithm 702 can continue examining the content of the interface signal.

[0103] In another approach (also called "state-based detection"), the extraction algorithm 702 can extract a machine state and then determine whether the machine state matches an associated state for data transfer. One example of an associated state is reaching a predefined interface (e.g., a checkout interface). Another example of an associated state is interacting with a source electronic device for at least a predefined amount of time. If the machine state matches the associated state, the extraction algorithm 702 can output a context tag that represents the machine state. For example, the context tag can include or imply information about the machine state. If the machine state does not match the associated state, the extraction algorithm can continue to examine the machine state.

[0104] FIG. 7B includes a schematic diagram illustrating how the detection algorithm 704 can examine the output generated by the extraction algorithm 702 to determine whether it is appropriate to initiate a transfer of information from a source electronic device to a recipient electronic device. The context tags generated by the extraction algorithm 702 as output can initially be provided as input to the detection algorithm 704. The detection algorithm 704 can then examine the context tags. For example, the detection algorithm 704 can compare each context tag to a predefined list of context tags to determine whether there is a match. The predefined list can include those context tags for which relevant information, such as advertisements, promotional offers, etc., is currently available. If the context tag matches an entry in the predefined list, the detection algorithm 704 can output the matching context tag. By outputting the matching context tag, the detection algorithm 704 can initiate a data transfer process, as further described below. Conversely, if the context tag does not match any entry in the predefined list, the detection algorithm 704 can discard the context tag and initiate the matching process with a new context tag.

[0105] FIG. 7C includes a schematic diagram illustrating how the modulation algorithm 706 can modulate the interface signal to overlay relevant information thereon. For purposes of explanation, the interface signal can be described as a "first signal." The context tag generated by the detection algorithm 704 as an output can initially be provided as an input to the modulation algorithm 706. For each context tag, the modulation algorithm 706 can determine whether there is a second signal in the signal database that represents the data to be transferred for that context tag. Each signal included in the signal database can represent one or more data packets that convey information about the corresponding context tag. The signal database can be the signal database 216 of FIG. 2, the signal database 312 of FIG. 3, or the signal database 408 of FIG. 4. If the second signal is in the signal database, the modulation algorithm 706 can obtain or extract the second signal from the signal database. However, if the second signal is not in the signal database, the modulation algorithm 706 can generate a second signal that represents the data to be transferred.

[0106] The modulation algorithm 706 can then ascertain the nature of the first signal. For example, the modulation algorithm 706 may determine whether the first signal is an audio signal or a video signal based on an analysis of its content, metadata, or format. If the first signal is an audio signal, the modulation algorithm 706 can generate an appropriate audio modification that encodes the content of the second signal that corresponds to the context tag. The audio modification may be generated based on an acoustic data transmission scheme, as described above. If the first signal is a video signal, the modulation algorithm 706 can generate an appropriate video modification that encodes the content of the second signal that corresponds to the context tag. The video modification may be generated according to an existing modulation scheme (e.g., IEEE 802.15) or a proprietary modulation scheme. In some embodiments, these video modulations are transposed by the modulation algorithm 706 to the mmWave domain such that these video modulations are detectable by a mmWave radar system. As an output, the modulation algorithm can generate a second signal including the data to be transferred, modulated according to the appropriate modulation scheme.

[0107] 7D includes a schematic diagram illustrating how the superposition algorithm 708 can superpose the second signal output by the modulation algorithm 706 onto the first signal broadcast by the source electronic device. As inputs, the superposition algorithm 708 can take (i) the first signal broadcast by the source electronic device and (ii) a second signal containing information to be communicated to the recipient electronic device. As discussed above with reference to FIG. 7C, the second signal can be modulated according to an appropriate modulation scheme based on the nature of the first signal.

[0108] Initially, the overlay algorithm 708 can extract equipment properties from the database. These equipment properties may relate to the source electronic device, the recipient electronic device, or both electronic devices. By examining the equipment properties, the overlay algorithm 708 can determine or infer whether the recipient electronic device can detect an audio or video signal broadcast by the source electronic device. Additionally, the overlay algorithm 708 can extract usage context properties from the database. Examples of usage context properties include a typical distance between the source electronic device and the recipient electronic device, a typical duration of time that the recipient electronic device spends in proximity to the source electronic device, etc. In some embodiments, the database is maintained on the source electronic device, while in other embodiments, the database is maintained on another electronic device (e.g., a computer server) to which the source electronic device is communicatively connected.

[0109] The superposition algorithm 708 may then combine the first and second signals to form a third signal. In some embodiments, this third signal is further manipulated and is therefore often referred to as a "prototype third signal." After forming the prototype third signal, the superposition algorithm 708 may estimate the detectability of the second signal based on the equipment properties and usage context properties extracted from the database. The detectability may depend on (i) parameters of the source electronic device, such as display size and audio volume, (ii) parameters of the recipient electronic device, and (iii) environmental factors, such as distance, background noise, etc. If the detectability of the second signal is determined to be low, the superposition algorithm may modulate the first and second signals to increase the detectability. Additionally, the superposition algorithm 708 may determine the degree to which the second signal is sensorily perceptible within the prototype third signal, taking into account the equipment properties and usage context properties. If it is determined to be likely sensorily perceptible, the superposition algorithm 708 may modulate the second signal to reduce noticeability. The conspicuousness of the second signal can be reduced, for example, by reducing the presence or magnitude of frequencies in the audible range, reducing contrast differences in visual content, or increasing the flicker rate above the visible range. As shown in FIG. 7D, the superimposition algorithm 708 can then determine whether the prototype third signal still meets the detectability requirement. If the prototype third signal still meets the detectability requirement, the superimposition algorithm 708 can output the third signal in a finalized form in preparation for broadcast by the source electronic device. This third signal can include data regarding the corresponding context tag as well as be presented alongside the first signal in a manner that is detectable by the recipient electronic device but minimally detectable (or not detectable at all) by the user of the source electronic device. If the prototype third signal does not meet the detectability requirement, the superimposition algorithm 708 can continue the modulation process as shown in FIG. 7D. [Illustrative Use Cases]

[0110] 8-9 show two different scenarios in which the approaches described herein may be implemented to transfer relevant information to an individual (and more specifically, to their electronic device) through modulation of an audible or visual signal.

[0111] FIG. 8 is a diagram illustrating how links may be intelligently embedded in a speech signal based on an analysis of the speech contained therein. One possible implementation of the approach described herein involves creating a speech embedding within the speech that serves as a "payment link." By recognizing relevant words (e.g., "amount," "dollars," etc.) and phrases (e.g., "transaction complete," "payment accepted," etc.), and then broadcasting the payment related information associated with those "cures" as metadata for each associated word, the payment related information may be embedded, possibly in real time, into the user's speech. This allows for advertisements to be generated in near real time within an individual's speech by recognizing words and phrases that constitute marketing or advertising opportunities.

[0112] As shown in FIG. 8 , the system 800 may include an interface signal (here, a live broadcast recording 802), a database 804 of information regarding marketing opportunities such as advertisements, and a detection algorithm 806. In this scenario, the detection algorithm 806 may be responsible for transcribing, establishing, or understanding the content of the live broadcast recording 802. In this manner, the detection algorithm 806 may be referred to as a “transcription algorithm.” The detection algorithm 806 may then determine whether any marketing opportunities contained in the database are suitable or appropriate given the content of the live broadcast recording. In other words, the detection algorithm 806 can determine whether the content “matches” any marketing opportunities contained in the database. A match may be determined based, for example, on the characteristics or meaning of words or phrases, user preferences, attributes of potential viewers, etc. Upon a match, the appropriate information may be forwarded to a broadcast system responsible for outputting the live broadcast recording 802.

[0113] To help illustrate the adaptability of the approach described herein, some additional example scenarios are provided below. As one example scenario, assume that an influencer is currently engaged in a live broadcast. In such a scenario, the system 800 can detect the product that the influencer is talking about. Once a "target word" is detected by the system 800, the system 800 can superimpose an ultrasound data packet with information about the product onto the next word or phrase. Every time someone records a live broadcast, that person can obtain the associated payment details. Additionally, every time a link is opened or a product is purchased via a link, the influencer can receive a reward. As another example, in a podcast, the system 800 can superimpose various advertisements onto words or phrases that match marketing opportunities in a category selected by the user. In a previously recorded podcast, updated marketing opportunities can be used to embed newer advertisements into the live broadcast.

[0114] FIG. 9 illustrates how the approach described herein can be used to facilitate secure transfer of sensitive information. More specifically, FIG. 9 illustrates how information related to (e.g., necessary for) a payment transaction can be visually communicated through modulation of a visual signal. An interesting application of the approach described herein includes digital receipts. However, the approach described herein can also be used to facilitate payment beacons, special offers and promotions, advertising, and the like.

[0115] For example, assume that an individual completes a payment transaction using a legacy system, such as an advertising system or a payment system. The legacy system may seamlessly communicate payment-related, but not overly sensitive, data to the recipient electronic device. This may be accomplished by embedding visual codes in its display. These visual codes may be examined, analyzed, or detected by sensors (e.g., a millimeter wave radar system) built into the recipient electronic device. The received information may be used by another computer program running on the recipient electronic device to provide features such as digital receipts, product information, advertisements, and special offers.

[0116] These visual codes may be embedded in the underlying video signal in the form of interlaced lines whose frequency encodes data, but may be too fast to be noticed by a person viewing the content of the display. Overall, this approach places low demands on legacy systems, so the approach described herein can be implemented simply by modifying the firmware or software that manages the display of the content.

[0117] It should be noted that the recipient electronic device may identify the moment these visual codes may be present (e.g., based on location) and then switch on its detection. In embodiments where detection is accomplished using a millimeter wave radar system, the user may not need to remove the recipient electronic device from his / her pocket, bag, etc. Instead, the millimeter wave radar system may be able to detect these visual codes through clothing. UI functions may also be managed based on the visual codes present. For example, UI functions may be enabled via the angle dependency of the code signal and the alignment of the recipient electronic device with the source electronic device.

[0118] As shown in FIG. 9, the source electronic device 902a-b can first display one or more digital images. An algorithm implemented either internally or externally can generate, for example, a "flash code" that carries information suitable to the content of these digital images. When the recipient electronic device 904a-d detects a situation in which a flash code may be present (e.g., based on a user being located near the source electronic device), it switches on its millimeter wave radar system. Alternatively, the recipient electronic device can actively scan nearby visual codes at a predetermined frequency. The data represented by the flash code can be read by the recipient electronic device via the millimeter wave radar system. The individual associated with the recipient electronic device is then presented with the results of the data capture (e.g., a digital receipt or advertisement).

[0119] In FIG. 9, two different use cases are illustrated. The first use case concerns special offers via an advertising interface (also called an "advertising screen"). For example, assume that an individual passes an advertising screen 902a during their daily travels in public transport (e.g., subway or bus). This advertising screen 902a may represent a source electronic device. A recipient electronic device 904a associated with the individual may actively scan the surrounding environment every few seconds to identify the presence of a visual code. In the public transport, the content of the source electronic device 902a is modulated such that a flash code related to the displayed advertisement can be embedded in the video signal. The advertisement may prompt nearby individuals to check their respective recipient electronic devices 904a-c for a surprise discount or promotion of a product offered. As mentioned above, sensors such as millimeter wave radar sensors can detect the modulation of the video signal without having to be removed from pants, jackets, etc. Thus, without having to be removed from a pocket, the recipient electronic device 904a can detect information from the flash code and store the information for subsequent use (e.g., if the information is related to an offer or promotion). The individual may then check a computer program (eg, a payment application) running on the recipient electronic device to see if the information is still usable.

[0120] A second use case relates to digital receipts for legacy payment systems. For example, assume that an individual with access to a mobile phone 904d is shopping at a store. And assume that a computer program running on the mobile phone 904d that allows for digital payments to be made has enabled a read function. This computer program may be referred to as a "payment application." In such a scenario, when the individual is determined to be in proximity to the payment system, the payment application may initiate a detection function supported by a millimeter wave radar sensor included in the mobile phone 904d. When the individual's transaction is processed by the payment system, the modified firmware may have it embedded with a code representing the items purchased, the transaction amount, and a timestamp. The individual may complete the transaction using a payment card (e.g., a credit card or debit card) that is programmatically associated with the payment application. The visual code output by the source electronic device 902b may be read by the recipient electronic device 904b. Furthermore, the payment application may be able to match the payment of their expenditures to the item list and time code, even if the recipient electronic device detects multiple different data packets while in the vicinity of the payment system. Thus, an individual may be provided with a digital receipt. [Processing system]

[0121] 10 is a block diagram illustrating an example of a processing system 1000 capable of implementing at least some of the operations described herein. For example, the components of the processing system 1000 may be hosted on an electronic device that includes a video or audio output component. As another example, the components of the processing system 1000 may be hosted on an electronic device that includes an operating platform capable of operating interface signals that are audibly or visually detectable by another electronic device for purposes of information transfer.

[0122] The processing system 1000 may include a processor 1002, a main memory 1006, a non-volatile memory 1010, a network adapter 1012 (e.g., a network interface), a video display 1018, an input / output device 1020, a controller 1022 (e.g., mechanical input such as a keyboard, pointing device, or buttons), a drive unit 1024 including a recording medium 1026, or a signal generator 1030, all communicatively coupled to a bus 1016. The bus 1016 is illustrated as an abstraction representing one or more physical buses and / or point-to-point connections connected by appropriate bridges, adapters, or controllers. Thus, the bus 1216 may be any of a variety of buses, including a system bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express bus, a HyperTransport bus, an Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an I / O bus, a USB bus, a USB 2.0 bus, a USB 3 ... 2 C (Inter Integrated Circuit) bus, or a bus conforming to IEEE standard 1394.

[0123] Processing system 1000 may share a similar computer processor architecture as that of a computer server, a router, a desktop computer, a tablet computer, a mobile phone, a video game console, a wearable electronic device (e.g., a watch or fitness tracker), a network-connected ("smart") device (e.g., a television or home assistant appliance), an augmented or virtual reality system (e.g., a head-mounted display), or another electronic device capable of executing (sequentially or otherwise) a set of instructions that specify the action(s) to be performed by processing system 1000.

[0124] Although main memory 1006, non-volatile memory 1010, and storage medium 1024 are depicted as a single medium, the terms "storage medium" and "machine-readable medium" should be interpreted to include a single medium or multiple media that store one or more sets of instructions 1026. Also, the terms "storage medium" and "machine-readable medium" should be interpreted to include any medium capable of storing, encoding, or carrying a set of instructions for execution by processing system 1000.

[0125] Generally, the routines executed to implement embodiments of the present disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as a "computer program"). A computer program typically comprises one or more instructions (e.g., instructions 1004, 1008, 1028) stored at various times in various memories and storage devices within a computing device. When read and executed by processor 1002, the instructions cause processing system 1000 to perform operations for implementing various aspects of the present disclosure.

[0126] Although the embodiments have been described in the context of a fully functional computing device, those skilled in the art will appreciate that various embodiments can be distributed as a program product in various forms. The present disclosure applies regardless of the particular type of machine or computer readable medium used to actually achieve the distribution. Further examples of machine and computer readable media include volatile memory devices, non-volatile memory devices 1010, removable disks, hard disk drives, recordable type media such as optical disks (e.g., compact disk read-only memory (CD-ROM) and digital versatile disk (DVD)), cloud-based storage, transmission type media such as digital and analog communication links.

[0127] The network adapter 1012 enables the processing system 1000 to broker data in a network 1014 with entities external to the processing system 1000 through any communication protocol supported by the processing system 1000 and the external entity. The network adapter 1012 may include a network adapter card, a wireless network interface card, a switch, a protocol converter, a gateway, a bridge, a hub, a receiver, a repeater, or a transceiver including chips (e.g., enabling communication via Bluetooth, Wi-Fi, or NFC). [remarks]

[0128] The foregoing description of various embodiments of the claimed subject matter has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments have been selected and described in order to best explain the principles of the invention and its practical application so that others skilled in the relevant art can appreciate the claimed subject matter, various embodiments, and various modifications that are suitable for the particular application contemplated.

[0129] Although the detailed description describes certain embodiments and the best mode contemplated, no matter how detailed the detailed description appears, the technology can be implemented in many ways. The embodiments may vary considerably in their implementation details while being encompassed in this specification. Specific terms used when describing certain features or aspects of various embodiments should not be interpreted as meaning to be redefined herein to be limited to any particular characteristic, feature, or aspect of the technology with which the term is associated. In general, the terms used in the following claims should not be interpreted to limit the technology to the specific embodiments disclosed herein unless those terms are expressly defined herein. Thus, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of carrying out or implementing the embodiments.

[0130] The terminology used herein has been selected primarily for ease of reading and explanation. It has not been selected to define or enclose subject matter. Thus, it is intended that the scope of the technology be limited not by this detailed description, but rather by any claims that issue in an application based hereon. Thus, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology, which is defined in the following claims. [Industrial Applicability]

[0131] The present disclosure is applicable to wireless communication of information.

Claims

1. outputting, by the first electronic device, a first signal sensorily perceptible by a user; receiving, by the first electronic device, an input indicative of an instruction to communicate information regarding the content of the first signal to a second electronic device located proximate to the first electronic device; superimposing, by the first electronic device, a second signal indicative of information onto the first signal to generate a third signal; and outputting, by the first electronic device, the third signal for detection by the second electronic device; A method for providing the above.

2. The method of claim 1 , wherein the second signal is superimposed on the first signal in such a way that modulation of the first signal is sensorily imperceptible to the user.

3. The method of claim 1 , wherein the first signal represents an audio signal emitted by an audio output component of the first electronic device.

4. the second signal represents an ultrasonic signal inaudible to the user; The method of claim 3 , wherein the superimposing comprises encoding the second signal in a portion of the first signal specified by or determined based on the input.

5. The method of claim 1 , wherein the first signal represents a video signal to be displayed by a video output component of the first electronic device.

6. the superimposing comprises encoding the first and second signals in a form such that the encoded second signal is detectable by a radar system included in the second electronic device; 6. The method of claim 5, wherein the radar system includes: (i) a transmitter capable of generating electromagnetic waves greater than 24 gigahertz (GHz); (ii) a first antenna capable of transmitting electromagnetic waves greater than 24 GHz; and (iii) a second antenna capable of receiving electromagnetic waves greater than 24 GHz.

7. The overlapping step comprises: identifying a time interval for superimposing the second signal based on an analysis of the content of the first signal; combining the first signal and the second signal such that the second signal coincides with the time interval; The method of claim 1 , comprising:

8. The method of claim 1 , wherein (i) the content of the second signal, or (ii) the manner in which the second signal is superimposed on the first signal, uniquely associates the second signal with the second electronic device.

9. When executed by a processor of an electronic device, the electronic device is outputting a first signal that is audibly or visually detectable by a user; receiving input indicative of instructions to transmit information to another electronic device; superimposing a second signal indicative of information onto the first signal to generate a third signal; , and outputting the third signal for detection by the other electronic device; A program for causing an operation comprising:

10. The operation includes:

10. The method of claim 9, further comprising identifying a portion of the first signal onto which the second signal is superimposed by examining content of the first signal in real time.

11. The operation includes: determining a time interval for superimposing the second signal based on an analysis of the content of the first signal; and 10. The method of claim 9, further comprising: combining the first signal and the second signal such that the second signal coincides with the time interval.

12. the first signal represents a video signal to be displayed by a video output component of a first electronic device; said superimposing introducing a visual modulation into said video signal over a time interval; the visual modulation affects only a portion of each frame of the video signal that falls within the time interval; The program according to claim 9.

13. The operation includes: determining a modulation scheme for communicating the information to the other electronic device based on a characteristic of the electronic device; The program according to claim 9 , wherein the superimposing is performed in accordance with the modulation scheme.

14. The program of claim 13 , wherein the characteristic is a display size, a display type, a model, or an intended application.

15. The program product of claim 13 , wherein the modulation scheme specifies at least one characteristic of the second signal.

16. 10. The program of claim 9, wherein the second signal is one of a plurality of signals superimposed on the first signal and is therefore included in the third signal.

17. 17. The computer-readable medium of claim 16, wherein each signal of the plurality of signals corresponds to a different time interval during which the third signal is output for detection by the other electronic device.

18. outputting, by the first electronic device, a first stream of encoded data that is audibly or visually detectable by a user; determining, by the first electronic device, that information accessible to the first electronic device is to be shared with a second electronic device; generating, by the first electronic device, a second stream of encoded data indicative of the information; and transferring, by the first electronic device, the information to the second electronic device by overlaying the second stream of encoded data onto a portion of the first stream of encoded data; A method for providing the above.

19. the first stream of encoded data represents a video signal to be displayed by a video output component of the first electronic device; 20. The method of claim 18, wherein the information is conveyed through a flicker pattern or coloration introduced into the video signal that is imperceptible to the human eye.

20. Implementing an algorithm by the first electronic device to detect brand names, product names, or person names included in a predefined list; and by the first electronic device based on an output generated by the algorithm; detecting when one of the brand name, the product name, or the person's name is spoken or displayed in the first stream of encoded data; Further equipped with the determining is performed in response to the detecting; The method of claim 18, wherein the information is associated with a detected brand name, product name, or person name.

Images