Broadcasting contextual information through modulation of audio and video interfaces

Abstract

Presented here is an approach to transfer relevant information from a first electronic device (also called a "source electronic device") currently outputting an interface signal detectable by a second electronic device (also called a "recipient electronic device") via modulation of the interface signal. By such modulation, the relevant information is added to the interface signal, which is then forwarded to the recipient electronic device. This can be achieved by superimposing another signal representing the relevant information onto the interface signal. Thus, by superimposing a signal onto the interface signal output by the source electronic device, the relevant information can be transferred from the source electronic device to the recipient electronic device.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Application No. 63 / 167,461, filed on March 29, 2021, titled "Contextual Data Broadcasts Via Audio - Visual Interface Modifications", which is hereby incorporated by reference in its entirety.

[0002] (Technical Field) This disclosure relates to wireless communication of information, and more specifically, to an approach for wirelessly communicating information between electronic devices through modulation of audible or visually detectable content.

Background Art

[0003] With the development of the Internet of Things (IoT), electronic devices are becoming increasingly available, and individuals expect to always be connected to the Internet, electronic devices, and each other. Historically, this has been achieved through wireless network protocols collectively known as "Wi - Fi (registered trademark)". These wireless network protocols enable multiple nearby electronic devices to exchange data using radio waves.

[0004] Due to the demand for better connectivity, there is an increasing expectation that almost all electronic devices have the function of processing audio or video in real - time and performing local digital signal processing. With such functions in mind, companies have started developing new technologies to facilitate wireless interaction between electronic devices. One of these technologies is data - over - sound. Data - over - sound enables data exchange between a first electronic device (also called the "source electronic device") that includes an audio output component and a second electronic device (also called the "recipient electronic device") that includes an audio input mechanism.

[0005] At a high level, data over sound functions similarly to a quick response (QR) code, except that the data is transmitted over an acoustic channel. In practice, the data is encoded by a first electronic device into an acoustic signal, for example, as a series of tones, to form an "acoustic barcode." The first electronic device then emits an acoustic signal for reception by a second electronic device. Upon reception, the second electronic device demodulates the acoustic signal to decode the data encoded therein. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 includes a high-level diagram showing how relevant information can be superimposed on the interface signal output by the source electronic device.

[0007] [Figure 2] Figure 2 includes a high-level representation of a system that can be used to broadcast machine-interpretable encoded data from one electronic device to one or more other electronic devices.

[0008] [Figure 3] Figure 3 shows an example of an electronic device that can implement an operating platform designed to manipulate interface signals to transfer information to a receiver electronic device.

[0009] [Figure 4] Figure 4 includes a schematic diagram of a procedure for transferring information from a first electronic device (also called the "source electronic device") to a second electronic device (also called the "receiver electronic device") through the modulation of an audible or visual signal.

[0010] [Figure 5] Figure 5 includes a high-level diagram of communication between a source electronic device and a receiver electronic device.

[0011] [Figure 6] Figure 6 includes a flowchart of the process performed by the source electronic device to transfer information to the receiver electronic device.

[0012] [Figure 7A] Figure 7A includes a schematic diagram showing how the extraction algorithm can process interface signals to identify appropriate opportunities to transfer information to a receiver electronic device.

[0013] [Figure 7B] Figure 7B includes a schematic diagram showing how a detection algorithm can examine the output generated by an extraction algorithm to determine whether it is appropriate to initiate the transfer of information from a source electronic device to a receiver electronic device.

[0014] [Figure 7C] Figure 7C includes a schematic diagram showing how a modulation algorithm can modulate an interface signal in such a way that it overlays relevant information onto it.

[0015] [Figure 7D] Figure 7D includes a schematic diagram showing how the superposition algorithm can superimpose the second signal output by the modulation algorithm onto the first signal broadcast by the source electronic device.

[0016] [Figure 8] Figure 8 shows how a link can be intelligently embedded in an audio signal based on an analysis of the audio contained within the audio signal.

[0017] [Figure 9] Figure 9 illustrates how the approach described herein can be used to facilitate the secure transfer of sensitive information.

[0018] [Figure 10]FIG. 10 is a block diagram showing an example of a processing system that can implement at least a part of the operations described in this specification.

[0019] Various features of the techniques described herein will become more apparent to those skilled in the art upon consideration of the detailed description in conjunction with the drawings. The embodiments are shown in the drawings by way of example and not by way of limitation. The drawings depict various embodiments for purposes of illustration, and those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Accordingly, while specific embodiments are shown in the drawings, the technology is amenable to various modifications. DETAILED DESCRIPTION OF THE INVENTION

[0020] There are several advantages to transferring data by sound, a concept commonly referred to as "acoustic data transmission" or "data - over - sound." Acoustic data transmission can be achieved using audible sound signals, but typically it is achieved using inaudible sound signals (also referred to as "ultrasonic signals"). As such, acoustic data transmission may be hardly perceptible, if not completely imperceptible, to nearby people. Further, since acoustic connections rely on voice output and input components, many of the currently used electronic devices can participate in data - over - sound. Put simply, acoustic data transmission is compatible with electronic devices of various form factors, architectures, etc. As an example, ultrasonic signals are broadcast by a given electronic device as "advertising cookies" and can be collected and examined by the "always - on" voice input component of nearby electronic devices. Acoustic data transmission is also secure because unauthorized access requires the responsible party (also referred to as a "hacker") to be located near the electronic device from which the acoustic signal is emitted in order to access the data encoded therein.

[0021] However, acoustic data transmission has several drawbacks.

[0022] For example, while acoustic data transmission does not require a network like the internet, the implementation of the necessary framework generally requires a network connection. For instance, suppose data is transferred audibly from a source electronic device to a receiver electronic device. In such a scenario, the source electronic device may need a framework that specifies, for example, when or how to transfer the data to the receiver electronic device. The term "framework" can refer to a set of algorithms, heuristics, or rules that control or influence how acoustic data transmission is performed by the source electronic device. Generally, the framework is implemented in software.

[0023] However, many electronic devices are unable to initiate network connectivity. For example, despite the relatively rapid adoption of network-connected POS systems (also known as “payment systems”), many legacy payment systems remain in place. In fact, research shows that approximately 93% of payment systems deployed in the US in 2020 were not “cloud-enabled.” Similarly, the majority of legacy advertising systems are not “cloud-enabled.” However, these legacy payment and advertising systems typically include either an audio output component or a video output component.

[0024] As another example, acoustic data transmission may be impractical or impossible in certain situations. In particular, acoustic data transmission has historically been unsuitable for implementation with audible content streaming to one or more recipient electronic devices. For this reason, acoustic data transmission is difficult to implement in time-sensitive situations such as sending receipts for settlement transactions and delivering advertisements, because the data being transmitted is related to a given time (and therefore timeliness is essential).

[0025] Therefore, the approach described here involves transferring relevant information from a source electronic device currently outputting an interface signal to a receiver electronic device via modulation of the interface signal. Through such modulation, the relevant information is added to the interface signal and then transferred to the receiver electronic device. This can be achieved by superimposing another signal representing the relevant information onto the interface signal. Thus, by superimposing a signal onto the interface signal output by the source electronic device, the relevant information can be transferred from the source electronic device to the receiver electronic device.

[0026] Furthermore, as will be discussed later, the nature of the interface signal (and therefore the superimposed signal representing related information) may depend on the performance of the source electronic device and the receiver electronic device. For example, suppose the source electronic device includes (i) an audio output component that can output an audio signal and (ii) a video output component that can output a video signal. In such a scenario, the interface signal may represent an audio signal emitted by the audio output component, or the interface signal may represent a video signal displayed by the video output component.

[0027] Therefore, superimposed signals may be included in a manner that depends on the context and content of the interface signal, the nature of the source electronic device, or the nature of the receiver electronic device. In some embodiments, the relevant information is transmitted by designing an ultrasonic signal that encodes the relevant information in a manner that is audibly detectable by the receiver electronic device but not audibly detectable by the individual associated with the receiver electronic device. In other embodiments, the relevant information is transmitted by designing a video modification that encodes the relevant information in a manner that is visually or otherwise detectable by the receiver electronic device but not visually detectable by the corresponding individual. For example, the video modification may be designed and implemented so as to be detectable by a radar system included in the receiver electronic device. An example of a radar system is a millimeter-wave (mm-wave) radar system, which includes (i) a transmitter capable of generating millimeter-range (mm) electromagnetic waves, (ii) a first antenna capable of transmitting mm-range electromagnetic waves, and (iii) a second antenna capable of receiving mm-range electromagnetic waves. Generally, the terms “mm range” and “millimeter-wave range” refer to the portion of the electromagnetic spectrum between 24 gigahertz (GHz) and 300 GHz. A second relevant detection method is optical camera communication (OCC), in which the image modulation is designed according to one of the methods known in the art so that it is detectable by an image sensor included in an electronic device having a visual line of sight to a display on which the image modulation is presented.

[0028] In some embodiments, the content of the superimposed signal is determined based on a real-time analysis of the content of the interface signal. For example, the superimposed signal may include information about a product mentioned or indicated in the interface signal, or it may include a digital receipt for a settlement transaction facilitated or completed using the interface signal. In addition or alternatively, the timing of the superimposed signal may be determined based on a real-time analysis of the content of the interface signal. In other words, the portion of the interface signal on which the superimposed signal is superimposed may depend on the content of the interface signal. For example, if the superimposed signal includes information about a product mentioned or indicated in the interface signal over a time interval, the superimposed signal may be coupled with the interface signal over that time interval to allow access to the information when the product is mentioned or indicated. As another example, if the superimposed signal includes a digital receipt for a settlement transaction facilitated or completed using the interface signal, the superimposed signal may be coupled with the interface signal in response to a determination, based on the analysis of the interface signal, that the settlement transaction has been completed.

[0029] The superimposed signal can also be uniquely linked to a recipient electronic device (and therefore to the individual associated with that recipient electronic device). The superimposed signal can be uniquely linked to a recipient electronic device through other characteristics such as timing, content, and location (determined based on, for example, coordinates of a global positioning system, an Internet Protocol address, etc.). Thus, (i) the content of the first signal, (ii) the content of the second signal, or (iii) the way in which the second signal is superimposed on the first signal uniquely associates the second signal with the recipient electronic device. For example, suppose a source electronic device is tasked with transferring information to multiple recipient electronic devices. In such a scenario, the information associated with each recipient electronic device can be superimposed on different parts of the interface signal. This allows the source electronic device to sequentially transmit information to multiple recipient devices.

[0030] The reception of interface signals by a receiver electronic device and the subsequent decoding of superimposed signals can enable other functionalities. For example, certain user interface (UI) functions may be made accessible for interaction on the receiver electronic device for a certain period of time (e.g., as long as the superimposed signal is detectable by the receiver electronic device). In this way, superimposed signals can function as beacons that enable the provision of location-based contextualized data.

[0031] For illustrative purposes, embodiments may be described in the context of transferring relevant information within a given context. For example, embodiments may be described in the context of supplementing advertisements with relevant information or providing digital receipts for settlement transactions. However, the approaches described herein may also be applicable to other contexts that benefit from wireless transmission of data from one electronic device to another.

[0032] While not mandatory, implementation is described below in the context of instructions executable by electronic devices. The term “electronic device” is generally used interchangeably with the term “computing device” and may therefore be used to refer to computer servers, payment systems, advertising systems, tablet computers, mobile phones, wearable devices (e.g., fitness trackers and watches), etc.

[0033] While certain aspects of this technology, such as specific modules, can be described as being executed exclusively or independently by a single electronic device, some implementations run in a distributed environment where modules are shared among multiple electronic devices linked via a network. For example, an individual may indicate that they require additional information about a product displayed by an advertising system. In such a scenario, the advertising system may retrieve information from a computer server accessible via a network (e.g., the Internet) and then transfer the information to the individual's mobile phone as described above. Alternatively, the information may be stored on the advertising system, and therefore the advertising system may not need to be connected to a network. [term]

[0034] In this specification, any reference to “one embodiment” or “one embodiment” means that the features, functions, structures, or characteristics described are included in at least one embodiment of the technology. The appearance of such phrases does not necessarily refer to the same embodiment, nor does it necessarily refer to alternative embodiments that are mutually exclusive.

[0035] Unless the context clearly requires otherwise, the terms “equipped,” “equipped,” and “composed of” shall be interpreted in an inclusive sense (i.e., “including but not limited to”) rather than in an exclusive or exhaustive sense. Similarly, the term “based on” shall be interpreted in an inclusive sense, rather than in an exclusive or exhaustive sense. Therefore, unless otherwise specified, the term “based on” is intended to mean “based on at least partially.”

[0036] The terms “connected,” “joined,” and their variations are intended to include any connection or joining, directly or indirectly, between two or more elements. The connection / joining can be physical, logical, or a combination thereof. For example, objects may be electrically or communicatively joined to one another even if they do not share a physical connection.

[0037] The term "module" can refer to a software component, a firmware component, or a hardware component. A module is typically a functional component that produces one or more outputs based on one or more inputs. For example, a computer program may contain multiple modules responsible for completing different tasks, or a single module responsible for completing all tasks.

[0038] When used in relation to a list of multiple items, the term "or" is intended to cover all interpretations: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0039] The order of steps performed in any of the processes described herein is illustrative. However, steps can be performed in various orders and combinations, as long as it does not conflict with physical feasibility. For example, steps can be added to or removed from the processes described herein. Similarly, steps can be replaced or their order can be changed. Thus, any description of a process is intended to be open-ended. [Introduction to Information Transmission through Signal Manipulation]

[0040] Figure 1 includes a high-level diagram showing how relevant information may be superimposed on the interface signal output by the source electronic device 102. As described above, the interface signal may be the audio signal 104 output by the audio output component 106, or the interface signal may be the video signal 108 output by the video output component 110.

[0041] At a high level, the approach introduced here allows relevant information to be superimposed on the interface signal so that it is transferred to the receiver electronic device 112. The source electronic device 102 and the receiver electronic device 112 may have network connectivity (e.g., via their respective antennas), but it may not be possible for these devices to connect to each other directly. In addition or alternatively, the source electronic device 102 may not be able to determine which electronic device is located, if any, within a predetermined proximity range to enable a link over a wireless communication channel. The approach presented here allows relevant information to be transferred from the source electronic device 102 to the receiver electronic device 112 even in such a scenario.

[0042] As will be discussed later, the context detection algorithm 114 (or simply the “detection algorithm”) may be responsible for detecting the context of the content of the interface signal. Based on the context, relevant information to be broadcast to the receiver electronic device 112 can then be identified. Generally, the content is analyzed locally (i.e., on the source electronic device 102) by the detection algorithm 114, although the content may be examined elsewhere by the detection algorithm 114. For example, in some embodiments, the detection algorithm 114 may be executed by another electronic device (e.g., a computer server) to which the source electronic device is communicably connected.

[0043] Subsequently, an appropriate modulation designed to transmit the relevant information can be identified by the superposition generation system 116. As will be described later with reference to Figure 2, the superposition generation system 116 may include a modulation algorithm, a superposition algorithm, a signal database, or any combination thereof. An appropriate modulation may be designed to be superimposed on an interface signal so that the relevant information is broadcast to nearby electronic devices such as the receiver electronic device 112. For example, the relevant information may be broadcast via the modulation of the audio signal 104 so that the modulation is detectable by the audio input component. As another example, the relevant information may be broadcast via the modulation of the video signal 108 so that the modulation is detectable by the video input component or other sensors (e.g., a millimeter-wave radar system).

[0044] For the sake of explanation, modulation is represented as a binary string, i.e., 0101001101, which can be superimposed on the interface signal to generate a manipulated signal. By superimposing modulation onto the audio signal 104, the superposition generation system 116 can generate a manipulated audio signal 118 that can be broadcast for detection by the receiver electronic device 112. Similarly, by superimposing modulation onto the video signal 108, the superposition generation system 116 can generate a manipulated video signal 120 that can be broadcast for detection by the receiver electronic device 112.

[0045] Typically, the superimposed generation system 116 manipulates only one of the audio signal 104 and the video signal 108. However, there may be scenarios in which the superimposed generation system 116 manipulates both the audio signal 104 and the video signal 108. For example, if an operating platform, of which the detection algorithm 114 and the superimposed generation system 116 are part, is not certain that the receiving electronic device 112 can detect either of these signals, the manipulated audio signal 118 and the manipulated video signal 120 may be broadcast for detection by the receiving electronic device 112. As another example, if the operating platform is interested in splitting the relevant information (e.g., for privacy purposes, or due to bandwidth or delay of those channels), the manipulated audio signal 118 and the manipulated video signal 120 may be broadcast for detection by the receiving electronic device 112. In embodiments in which the manipulated audio signal 118 and the manipulated video signal 120 are broadcast for detection, those signals may contain the same information or different information.

[0046] As shown in Figure 1, the receiver electronic device 112 can detect the manipulated signal broadcast by the source electronic device 102. If the manipulated signal is a manipulated audio signal 118, the receiver electronic device 112 may detect the manipulated signal via an audio input component. If the manipulated signal is a manipulated video signal 120, the receiver electronic device 112 may detect the manipulated signal via a video input component or other sensors (e.g., a millimeter-wave radar system capable of sensing the state of the video screen). The data can then be decoded from the manipulated signal by the receiver electronic device. Based on this data and other features (e.g., timing and location), relevant information may be provided to the user of the receiver electronic device 112. Examples of relevant information include personalized offers from brands, products, and merchants, and digital receipts for payment transactions. [Overview of the signal manipulation platform]

[0047] Figure 2 includes a high-level representation of System 200 that can be used to broadcast machine-interpretable encoded data from electronic device 204 to one or more other electronic devices (not shown). For illustrative purposes, electronic device 204 may be referred to as the “source electronic device,” and the other electronic devices may be referred to as “receiver electronic devices.” To broadcast machine-interpretable encoded data, a signal manipulation platform 202 (or simply the “manipulation platform”) can manipulate interface signals output by electronic device 204. For example, the manipulation platform 204 may manipulate video signals displayed by the video output component 206 of electronic device 204, or it may manipulate audio signals emitted by the audio output component 208 of electronic device 204. These manipulations may be designed and implemented such that the resulting changes in the video or audio signals are not entirely, if not entirely, perceptible to individuals associated with the receiver electronic devices. These individuals may be referred to as “users” of the receiver electronic devices.

[0048] It should be noted that the term "sensory perceptible" can refer to a stimulus that can be seen, noticed, or detected by typical human senses. A stimulus may be visually perceptible, auditorily perceptible, tactilely perceptible, etc. Examples of visual stimuli include still and moving images, while examples of audible stimuli include audiobooks, podcasts, and radio broadcasts. Some stimuli may be perceptible by multiple senses (for example, a video signal may be visually and auditorily perceptible). Therefore, the term "sensory inperceptible" may be used to refer to a stimulus that cannot be seen, noticed, or detected by typical human senses.

[0049] For example, suppose an individual visits a retail establishment where audible content is being emitted. As the individual walks through the retail establishment, the underlying audio signal may be manipulated to include information about products offered by the retail establishment or about the retail establishment itself. For example, the information could be a hyperlink (or simply a “link”) to a website containing details about products or promotional campaigns offered by the retail establishment. This would involve manipulating the underlying audio signal, but the manipulation may not be detectable by the human ear. Therefore, an electronic device may detect the manipulation, but the individual may not realize that the underlying audio signal has been manipulated. By decoding the manipulated audio signal, the electronic device can “discover” the information. Similar processes for transmitting information via modulation of audible content can occur when an individual is listening to a podcast at home, listening to the radio in a car, or walking through a public environment such as a shopping center or airport.

[0050] As another example, suppose an individual is walking through a public or private environment while wearing an augmented reality headset (also called a “mixed reality headset”). As the individual walks through the environment, they may observe display panels that visually present content related to products, people, events, etc. Furthermore, as will be discussed later, the visual content displayed on these display panels may be manipulated to include information about those products, people, events, etc. For example, if the content is about a music performance by a certain artist, the information may be a link to a website where tickets for that performance can be purchased. As another example, if the content is about a movie showing in a cinema, the information may be a link to a trailer for that movie. Again, the manipulation causes a change in the underlying video signal, but the manipulation may not be detectable by the human eye. Thus, the mixed reality headset may detect the manipulation, but the individual may not realize that the underlying video signal has been manipulated. By decoding the manipulated video signal, the mixed reality headset can “discover” the information. Equivalent processing may be performed by other types of electronic devices, including sensors (e.g., image sensors) that can visually monitor the environment. Examples of such electronic devices include mobile phones, wearable cameras, and some watches and fitness accessories. Specifically, a mobile phone may be able to detect changes in visual content when panned across an environment that includes its display panel.

[0051] As shown in Figure 2, the system 200 includes an operating platform 202 that can access a video output component 206, an audio output component 208, a processor 210, or any combination thereof. Those skilled in the art will recognize that not all of these components necessarily need to be included in the electronic device 204. For example, in some embodiments, the electronic device 204 may include an audio output component 208 but not a video output component 206. In other embodiments, the electronic device 204 may include a video output component 206 but not an audio output component 208. In addition to the components shown in Figure 2, the electronic device 204 may also include other components. For example, in addition to the audio output component 208, the electronic device 204 may include an audio input component to enable audio interaction between the electronic device 204 and a receiving electronic device or the user of such a device. An example of an audio input component is a microphone designed to convert sound into electrical impulses to create a signal that can be processed by other components of the electronic device 204.

[0052] The video output component 206 can be any mechanism capable of operating to visually transmit the content of a video signal to an individual and a corresponding receiver electronic device. For example, the video output component 206 may be a display panel including light-emitting diodes (LEDs), organic LEDs, liquid crystal elements, or electrophoretic elements.

[0053] The audio output component 208 can be any mechanism capable of operating to audibly transmit the content of an audio signal to an individual and a corresponding receiver electronic device. For example, the audio output component 208 could be a loudspeaker (or simply a “speaker”) designed to convert electrical impulses into sound.

[0054] The electronic device 204 can output an audio signal through the audio output component 208 and a video signal through the video output component 206, and therefore may be called an “audio-video device” or “audio-visual device.” As described above, during operation, the electronic device 204 may output an interface signal detectable by a user. In embodiments where the interface signal represents an audio signal, the electronic device 204 may be described as generating or supporting an “audio interface” with which a user can interact. In embodiments where the interface signal represents a video signal, the electronic device 204 may be described as generating or supporting a “video interface” or “visual interface” with which a user can interact. In some scenarios, the electronic device 204 may output multiple interface signals, such as an audio signal output by the audio output component 208 and a video signal output by the video output component 206. In such embodiments, the electronic device 204 may be described as generating or supporting an “audio-video interface” or “audio-visual interface” with which a user can interact. Furthermore, as will be described later, the operating platform 202 can control one or both of these interface signals.

[0055] As shown in Figure 2, the operating platform 202 may include a signal modulation algorithm 212 (or simply the “modulation algorithm”), a signal superposition algorithm 214 (or simply the “superposition algorithm”), and a signal database 216. The signal database 216 may store various signals representing information that can be shared with the receiver electronic device. In other words, the signal database 216 may store various signals that can be superimposed on the interface signal output by the electronic device 204. Furthermore, as will be discussed later, the signals in the signal database 216 may vary depending on the nature of the interface signal output by the electronic device 204, the components of the electronic device 204, or the components of the receiver electronic device. Therefore, the signal database 216 may include acoustic signals, video signals, or a combination thereof. The signals in the signal database 216 may be encrypted, hashed, or obfuscated to prevent unauthorized access.

[0056] In some embodiments, the operation platform 202 is configured to generate signals superimposed on interface signals in near real time. For example, the operation platform 202 may generate signals superimposed on interface signals based on a real-time analysis of the content of interface signals. As another example, the operation platform 202 may generate signals superimposed on interface signals based on one or more characteristics of a receiver electronic device (e.g., learned through analysis of communications transmitted from the receiver electronic device to the electronic device 204 in the form of one or more data packets). In embodiments where signals are generated by the operation platform in near real time, the signals may not be stored in the signal database 216 at all, or they may be stored in the signal database 216 only for a short time (e.g., a few seconds or less) until the transfer of information is complete.

[0057] When executed by processor 210, the algorithm implemented on the operation platform 202 enables the transfer of information from electronic device 204 to receiver electronic device through the manipulation of an audibly or visually perceptible interface signal. To achieve this, the operation platform 202 can complete two phases, each corresponding to a different algorithm, as shown in Figure 2.

[0058] In the first phase (also called the “modulation phase” or “creation phase”), the modulation algorithm 212 can create a signal that represents information to be shared with the receiver electronic device. This can occur in various ways. In some embodiments, the modulation algorithm 212 retrieves a signal from the signal database 216 based on the content of the interface signal or the characteristics of the receiver electronic device or its user. This signal can be used without any modulation (i.e., in its “raw form”), or it can be modulated by the modulation algorithm 212 to be tailored to the interface signal, the receiver electronic device, or the user of the receiver electronic device. In other embodiments, the modulation algorithm 212 generates a signal in near real time based on an analysis of the interface signal on which the signal is superimposed.

[0059] In the second phase (also called the "superposition phase" or "transfer phase"), the superposition algorithm 214 can superimpose the signal onto the interface signal so that the information to be transferred to the receiver electronic device is "carried together" with the interface signal. The superposition can be performed such that the interface signal is subjected to minimal degradation. Therefore, the modulation resulting from the superposition of the signal onto the interface signal may be almost imperceptible, if not completely imperceptible, to the user of the receiver electronic device, but is readily detectable by the receiver electronic device itself.

[0060] As described above, in some embodiments, the interface signal may represent an audio signal. Therefore, by superimposing a second audio signal (e.g., an ultrasonic audio signal) onto the audio signal functioning as the interface signal, information can be transferred from the electronic device 204 to the receiving electronic device. In such embodiments, the timing of information transfer may be optimized based on an analysis of the content contained in the interface signal (e.g., spoken language, non-textual (non-lexical) utterances). In other words, the portion of the interface signal onto which the second audio signal is superimposed can be determined based on an analysis of the content of the interface signal. In addition or alternatively, the portion of the interface signal may be specified by, or determined based on, an input received by the operating platform 202 indicating that information is being transferred. Those skilled in the art will recognize that in some situations, an interface signal may be accompanied by another interface signal. For example, the electronic device 204 may output a first interface signal containing audio content related to a product or service, and a second interface signal containing visual content related to the product or service. In such scenarios, the timing of information transfer may be optimized based on the content of the first interface signal, the content of the second interface signal, or a combination thereof.

[0061] In other embodiments, the interface signal represents a video signal. In such embodiments, the superposition of a second audio signal onto the video signal acting as the interface signal can be a visual modulation of the content. This visual modulation may be visible to the human eye, or it may not be perceptible to the human eye but may be detectable by a receiving electronic device. Examples of visual modulation include watermarks, flicker patterns, and color scheme changes. In embodiments where the interface signal represents a video signal, the timing of information transfer may also be determined based on an analysis of the content of the interface signal. For example, in response to a determination that the interface signal output by the video output component 206 contains a specific message (e.g., "Thank you for your purchase"), content (e.g., a website confirming that the payment transaction has been completed), etc., information may be transferred from the electronic device 204 to the receiving electronic device. Furthermore, as will be discussed later, various sensors included in the receiving electronic device can be used to detect, analyze, or understand the second video signal superimposed on the interface signal. For example, the operating platform 202 may be used to design appropriate video signal modulation to transfer information in a manner optimized for detectability via a millimeter-wave radar system.

[0062] Figure 3 shows an example of an electronic device 300 that can implement an operating platform 310 designed to manipulate interface signals to transfer information to a receiving electronic device. As described above, the interface signal may represent an audio or video signal that is modulated when another signal is superimposed thereon. The modulation is detectable by the receiving electronic device to which the information is transferred, but it does not necessarily have to be detectable by its user.

[0063] In some embodiments, the operation platform 310 is embodied as a computer program executed by an electronic device 300. For example, the operation platform 310 may reside on a payment system, in which case the modulation introduced into the interface signal may represent a digital receipt communicated to individuals who have completed a payment transaction using their respective recipient electronic devices. In another example, the operation platform 310 may reside on an advertising system, in which case the modulation introduced into the interface signal may represent a link to additional information about a product or service displayed on the advertising system. Those skilled in the art will recognize that embodiments of the operation platform 310 can be distributed among multiple electronic devices. For example, the operation platform 310 may reside on a first electronic device (e.g., a payment system, an advertising system, etc.), while the signal database 312 may reside on a second electronic device (e.g., a computer server).

[0064] The electronic device 300 may include a processor 302, memory 304, UI output mechanism 306, and communication module 308.

[0065] The processor 302 may have general-purpose characteristics similar to a general-purpose processor, or it may be an application-specific integrated circuit (ASIC) that provides control functions to the electronic device 300. As shown in Figure 3, the processor 302 may be directly or indirectly coupled to all components of the electronic device 300 for communication purposes. In embodiments in which the operating platform 310 is embodied as a computer program, underlying instructions may be executed by the processor 302 so that the electronic device 300 can perform the operations described herein.

[0066] Memory 304 may consist of any suitable type of storage medium, such as static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or registers. In addition to storing instructions that can be executed by processor 302, memory 304 may also store (i) data output by UI output mechanism 306, and (ii) data generated by processor 302 (e.g., when executing algorithms, heuristics, or rules of operation platform 310). Note that memory 304 is merely an abstract representation of the storage environment. Memory 304 may consist of an actual integrated circuit (also called a "chip") or module.

[0067] The communication module 308 may be responsible for managing communication between components of the electronic device 300. In addition or alternatively, the communication module 308 may be responsible for managing communication with other electronic devices (e.g., receiver electronic devices 314a-n). Thus, in some embodiments, the electronic device 300 can communicate with receiver electronic devices 314a-n via a network, while in other embodiments, the electronic device 300 cannot communicate with receiver electronic devices 314a-n via a network. Instead, the electronic device 300 may communicate with receiver electronic devices 314a-n via interface signals, as described above. Examples of receiver electronic devices 314a-n include mobile phones, tablet computers, personal computers, wearable devices, etc. On the other hand, the electronic device 300 may be a mobile phone, tablet computer, personal computer, wearable device, payment system, advertising system, etc. As an example, the electronic device 300 may be a payment system, while the receiver electronic devices 314a-n may be mobile phones associated with each user who completes a payment transaction using the payment system.

[0068] In embodiments where the communication module 308 can facilitate communication between the electronic device 300 and an external destination (e.g., receiver electronic devices 314a-n), the communication module 308 may be a wireless communication circuit designed to establish a communication channel with the destination. Examples of wireless communication circuits include chips configured for Bluetooth®, Wi-Fi, Near Field Communication (NFC), etc.

[0069] For convenience, the operating platform 310 may be referred to as a computer program residing in memory 304. However, the operating platform 310 may consist of software, firmware, or hardware implemented on or accessible from the electronic device 300. According to embodiments described herein, the operating platform 310 may include various algorithms as described above with reference to Figure 1-2. Typically, these algorithms are executed by separate modules of the operating platform 310 that are separately addressable (and therefore can run independently without interfering with other modules). These modules may be an integral part of the operating platform 310. Alternatively, these modules may be logically separate from the operating platform 310 but can operate "in parallel" with it. Together, these algorithms may enable the operating platform 310 to modulate interface signals to transfer information to a destination.

[0070] For example, suppose a person completes a payment transaction with an electronic device 300 that functions as a payment system. Traditionally, when a payment transaction is completed with a high-performance payment system, a digital receipt is provided in the form of a text message, email message, or push notification, and the underlying information is transmitted from these high-performance payment systems over a network. However, some payment systems cannot connect to a network such as the Internet. The approach described herein allows these payment systems to transfer information (e.g., a digital receipt) to a nearby recipient electronic device through the modulation of an audible or visual signal. In addition to allowing non-high-performance payment systems to transfer information quickly with little computational resource consumption, this approach is also secure. Therefore, this approach can be adopted by electronic devices even if the electronic device can connect to a network such as the Internet. As an example, the operating platform 310 may transfer information (e.g., a digital receipt) to a nearby recipient electronic device through the modulation of an audible or visual signal, instead of transferring information over a network, or in addition to doing so.

[0071] Other elements may also be included as part of the operating platform 310. The nature of these other elements may depend on the intended use of the electronic device 300 and the operating platform 310. For example, the UI module may be responsible for generating content output by the UI output mechanism 306 for presentation to one or more receiver electronic devices 314a-n. Generally, the content output by the UI output mechanism 306 is tailored to each user or each receiver electronic device. Thus, the UI output mechanism 306 may output a first interface signal for detection by receiver electronic device 314a, a second interface signal for detection by receiver electronic device 314b, and so on.

[0072] The form of the content may depend on the nature of the UI output mechanism 306. For example, if the UI output mechanism 306 is a speaker, the content of the interface signal may relate to audible confirmation of an event (e.g., completion of a payment transaction) or an audible description of an item (e.g., a product or service). As another example, if the UI output mechanism 306 is a display panel, the content of the interface signal may relate to visual confirmation of an event (e.g., completion of a payment transaction) or a visual description of an item (e.g., a product or service). Furthermore, as will be discussed later, the UI output mechanism 306 may also play a role in outputting (e.g., emitting or displaying) an interface signal on which another signal is superimposed in order to audibly or visually modulate the interface signal.

[0073] During operation, the operating platform 310, in combination with other components of the electronic device 300, may represent a system capable of analyzing the audible or visual content of an interface signal and embedding relevant data packets (e.g., links or offers) at moments related to the underlying content of the interface signal. In some embodiments, the operating platform 310 operates in near real-time to enable use with live media content. Examples of live media content include radio programs, live television programs, and live streaming events.

[0074] As an example, an extraction algorithm may be designed or trained to recognize potential audio features in speech (also called “sound tags”), such as references to brands, products, or attributes of brands or products, and to assign labels to speech to indicate these sound tags. Thus, different parts of audio content may be “tagged” or “labeled” to indicate their content. When the detection algorithm detects relevant sound tags, information about each sound tag may be transmitted to one or more nearby recipient electronic devices. For example, if the detection algorithm finds sound tags associated with a given product, the operation platform 310 (more specifically, its modulation algorithm and superposition algorithm) may transmit links to websites associated with the given product to one or more recipient electronic devices.

[0075] Alternatively, the audible or visual content of the interface signal may be “pre-analyzed” in static mode. In this scenario, the operating platform 310 can investigate the interface signal in a similar manner, but this investigation is not performed at runtime (e.g., while the interface signal is output for detection by one or more receiver electronic devices 314a-n). One advantage of “pre-analyzing” the interface signal is that less computational resources are required for labeling with sound tags because timeliness is less of a concern. Another advantage of “pre-analyzing” the interface signal is that more robust detection algorithms can be used that cannot be implemented in near real-time without delay. Similarly, “pre-analyzing” the interface signal may allow the operating platform 202 to perform a multi-stage analysis (e.g., with a detection algorithm that identifies the sound tag quickly but with relatively low accuracy, and another detection algorithm that identifies the sound tag slowly but with relatively high accuracy).

[0076] As will be discussed later, the operating platform 310 may be programmed to design and output visual modulation in addition to, or instead of, audible modulation. For example, the operating platform 310 may be able to design and output visual modulation optimized for detection by a millimeter-wave radar system. Visual modulation can be designed so that the modulation causes a large state change in the video output component (e.g., a display panel) without being noticed by the human eye. An example of suitable visual modulation is a flicker pattern, in which adjacent rows or columns of a display panel cycle between different color states, and the information is encoded via a suitable frequency modulation scheme. Visual modulation does not necessarily have to be visually detectable (e.g., detection by a millimeter-wave radar system), but may be easily detectable by a receiving electronic device, while the visibility to the corresponding user can be minimized by selecting a carrier frequency that exceeds the flicker fusion frequency for human vision. Alternatively, visual modulation may be noticeable to the user but affect only a portion of the interface signal. For example, visual modulation may be superimposed on less important parts of the video signal, such as the top or bottom corners. Therefore, modulation can only affect a portion of each frame contained within the interface signal on which the signal is superimposed.

[0077] As part of the information transfer process, the operating platform 310 may identify one or more characteristics of the electronic device 300. Examples of characteristics include display size, display type (e.g., LCD, OLED, etc.), model, desired detection distance from the receiver electronic device, intended application of the approach described herein, and nearby noise sources (e.g., radio frequency noise). Based on these characteristics, the operating platform 310 may identify a modulation scheme. The modulation scheme may govern how information is transferred to the receiver electronic device. The modulation scheme may specify the nature of the signal superimposed on the interface signal. For example, the modulation scheme may specify characteristics of the signal, such as whether the signal is an audio signal or a video signal, whether the signal is ultrasonic, or whether the video signal is intended to be detectable by a particular sensor (e.g., a millimeter-wave radar system). Each information transfer may be performed according to the modulation scheme, but the modulation scheme does not necessarily have to impose the same constraints (e.g., regarding packet size) on each receiver electronic device.

[0078] As described above, electronic device 300 may be tasked with the task of transmitting information to multiple recipient electronic devices 314a-n simultaneously or sequentially. Since the information transfer includes audible or visual signals, the same information may be transferred to each recipient electronic device that simultaneously "hears" or "sees" the interface signal. To transfer different information to each recipient electronic device, recipient electronic devices 314a-n may eliminate ambiguity in the transmissions through timing, content, and other characteristics. For example, suppose electronic device 300 functions as a payment system where different individuals complete payment transactions. Digital receipts for these payment transactions may be indiscriminately broadcast by electronic device 300. In such a scenario, recipient electronic devices 314a-n may "hear" many of these transmissions. For each transmission "hearsed," each recipient electronic device may search the computer program (e.g., payment application) running on that recipient electronic device to find a transaction in which the information matches one of the transmissions. This allows the receiving electronic device to associate the received data (e.g., purchased items and associated costs) with the corresponding user and provide a digital receipt. Remaining transmissions, if received as irrelevant, may be ignored or deleted. In some embodiments, transmissions may be distinguished based on timing. For example, the receiving electronic device may "search" for transmissions at regular intervals after determining that a transaction has been completed, based on information available from a computer program (e.g., a payment application).

[0079] The reception and subsequent decoding of interface signals by a receiver electronic device can enable a variety of functionalities. For example, a specific UI function may be made accessible for interaction on the receiver electronic device for a certain period of time (e.g., as long as the superimposed signal is detectable in the interface signal). Thus, the superimposed signal can function as a beacon, enabling the provision of location-based contextualized data. As another example, a specific UI function may be activated on the receiver electronic device based on its movement during the period of information transfer. For example, the UI may take the form of physical movement on the receiver electronic device relative to the electronic device 300 such that the interface signal on which another signal is superimposed is modified by its directionality and intensity. A specific action may be associated with a UI function such as accepting or rejecting a proposed course of action (e.g., transferring information, completing a settlement transaction). [Methodology of signal manipulation]

[0080] Figure 4 includes a schematic diagram of procedure 400 for transferring information from a first electronic device (also called the “source electronic device”) to a second electronic device (also called the “receiver electronic device”) through the modulation of an audible or visual signal. This audible or visual signal may be called the “interface signal” or “first signal”, detectable by the user of the source electronic device. As described above, the user may be associated with the receiver electronic device.

[0081] In some embodiments, the first signal is detectable by a receiving electronic device. For example, if the first signal includes human-audible audio content, the receiving electronic device may detect the first signal using an audio input component. However, the first signal does not necessarily have to be detectable by the receiving electronic device. For example, suppose a payment system transfers data containing information related to a payment transaction to a customer's mobile phone. In such a scenario, the payment system represents the source electronic device, while the mobile phone represents the receiving electronic device. The first signal could be human-readable video content displayed by the payment system. To transfer data (e.g., specifying purchased items and associated costs) to the mobile phone, a signal manipulation platform running on the payment system can modulate the first signal so that the modulation is detectable by sensors included in the mobile phone, even if the mobile phone is not visually observing the payment system at that moment. For example, the modulation of human-readable video content may be detectable by a millimeter-wave radar system included in the mobile phone. This example also demonstrates that the user of the source electronic device (e.g., a cashier) does not necessarily have to be the same person as the user of the receiving electronic device (e.g., a customer).

[0082] First, the operating platform can acquire a first signal. In some embodiments, the first signal is acquired in near real-time (for example, in conjunction with the output by the UI output mechanism 412). In other embodiments, the first signal is "pre-checked" before being output by the UI output mechanism 412, as described above. Next, the operating platform can provide the first signal as input to the extraction algorithm 402 to acquire a set of tags. If the first signal represents an audio signal, these tags may represent "sound tags". If the first signal represents a video signal, these tags may represent "video tags". Regardless of their nature, these tags may represent labels indicating the content of the first signal.

[0083] Next, the operating platform can provide the first signal and associated tags to the detection algorithm 404. When relevant tags are detected by the detection algorithm 404, information about those tags can be transferred to a nearby recipient electronic device. Thus, the detection algorithm 404 can parse the tags generated by the extraction algorithm 402 to identify an appropriate opportunity to transfer information to a recipient electronic device. As output, the detection algorithm 404 can generate a representation of the relevant tags. For example, the detection algorithm 404 may maintain an ordered list of tags representing or associated with products of interest for advertising purposes.

[0084] The display of relevant tags may be provided to the modulation algorithm 406 by the operating platform as input. As described above, the modulation algorithm 406 may be responsible for determining the information to be transferred to the receiver electronic device based on the relevant tags detected by the detection algorithm 404. This information may be represented as a second signal. In some embodiments, the second signal is retrieved from the signal database 408 by the modulation algorithm 406, as shown in Figure 4. Alternatively, the modulation algorithm 406 may generate a second signal based on tags deemed relevant by the detection algorithm 404.

[0085] Subsequently, the operating platform can provide the first and second signals as inputs to the superposition algorithm 410. As described above, the superposition algorithm 410 may superimpose the second signal onto the first signal to generate a third signal that can be output via the UI output mechanism 412 for detection by a receiver electronic device. At a high level, this third signal may represent the second signal combined with the first signal. Generally, the second signal is superimposed only on a portion of the first signal. For example, if the first signal represents an audio signal corresponding to recorded or streamed audio content, the second signal may be superimposed only on a few seconds of the audio signal. As another example, if the first signal represents a video signal corresponding to recorded or streamed video content, the second signal may be superimposed only on a few seconds of the video signal. Thus, the UI output mechanism 412 may output the third signal for a limited time (e.g., a few seconds or a few minutes), and the first signal may be output before and after the limited time.

[0086] Figure 5 includes a high-level diagram of the communication between the source electronic device 500 and the receiver electronic device 550. The source electronic device 500 may be the electronic device 204 in Figure 2 or the electronic device 300 in Figure 3. First, the source electronic device 500 can output a first signal that is (i) detectable by the receiver electronic device 550 and / or (ii) detectable by the user of the source electronic device 500 (step 501). As described above, the user of the source electronic device 500 may be associated with the receiver electronic device 550 or may be unrelated to the receiver electronic device 550. In some embodiments, the first signal represents an audio signal emitted by the audio output component of the source electronic device 500. In other embodiments, the first signal represents a video signal displayed by the video output component of the source electronic device 500.

[0087] Subsequently, the source electronic device 500 may receive an input indicating an instruction to transmit information to the receiver electronic device 550 (step 502). Generally, this input is generated or detected by a detection algorithm as described above with reference to Figures 1 and 4. If the context of the first signal output by the source electronic device 500 is determined to match one of a number of predetermined classes (also called “context categories”), the detection algorithm may instruct the signal manipulation platform to begin transmission. For example, if the source electronic device 500 is a payment system, one context category indicating that a transfer should occur may be reaching a checkout screen. As another example, if the source electronic device 500 is an advertising system, one context category indicating that a transfer should occur may be displaying content discussing brands, products, or merchants for which offers are available. The input may also represent the discovery of a tag of interest, as described above. For example, the input may represent a determination that a sound tag related to a brand, product, or merchant of interest has been detected. Alternatively, the source electronic device 500 may receive a communication from the receiver electronic device 550 indicating that information is desired. For example, the receiving electronic device 550 may "ping" the source electronic device 500 in response to a decision that information from the source electronic device 500 is desired (e.g., based on analysis of a first signal, input provided by the user, etc.).

[0088] The source electronic device 500 can acquire a second signal indicating information (step 503), and then superimpose the second signal onto the first signal to generate a third signal (step 504). As described above, the second signal is usually superimposed on a portion of the first signal. Thus, the source electronic device 500 can output the first signal as described above with reference to step 501, output a third signal for detection by the receiver electronic device (step 505), and then output the first signal again when the second signal is no longer superimposed thereon.

[0089] How the second signal is superimposed on the first signal may depend on the properties of the first signal, as well as the properties of the source electronic device 500 or the receiver electronic device 550.

[0090] In embodiments where the first signal represents an audio signal, the second signal may represent an ultrasonic signal that is audible to the receiving electronic device 550 but not to its user. In such scenarios, the second signal may be superimposed on the first signal by encoding the second signal onto a portion of the first signal that is specified by or determined based on the input. For example, superposition may be performed such that the second signal (e.g., representing a digital receipt) is encoded onto the first signal (e.g., representing a UI indicating a completed payment transaction) in a time-aware manner, so that the information encoded in the second signal is provided to the appropriate person.

[0091] In embodiments where the first signal represents a video signal, superposition may include encoding the first and second signals in such a way that the encoded second signal is detectable by a sensor included in the receiver electronic device 550. The sensor may be, for example, a millimeter-wave radar system designed for electromagnetic waves above 24 GHz.

[0092] As described above, the second signal may be superimposed on the first signal for a certain time interval. Based on an analysis of the content of the first signal, the operating platform may determine the time interval for which the second signal should be superimposed on the first signal. The operating platform can then combine the first and second signals so that the second signal, which represents the information to be transmitted, matches the time interval.

[0093] In summary, the source electronic device 500 can output a first signal that is detectable aurally or visually by the receiver electronic device 550 and / or the user of the source electronic device 500, can receive an input indicating an instruction to transmit information to the receiver electronic device 550, can superimpose a second signal indicating information onto the first signal to generate a third signal, and can output a third signal for detection by the receiver electronic device 550.

[0094] As described above, the source electronic device 500 may communicate with multiple receiver electronic devices in several scenarios. Therefore, aspects of procedure 500 may be performed by the source electronic device 500 simultaneously or sequentially to communicate with each of these receiver electronic devices. For example, steps 502-505 may be performed sequentially to transfer information to each of these receiver electronic devices. It should be noted that the information transferred to each of these receiver electronic devices does not need to be the same. Therefore, the second signal superimposed on the first signal may be different for each receiver electronic device.

[0095] Figure 6 includes a flowchart of a process 600 performed by a source electronic device to transfer information to a recipient electronic device. Initially, the source electronic device can output a first stream of encoded data that can be detected audibly or visually by a user (step 601). Next, the source electronic device can decide that the information accessible therein should be shared with the recipient electronic device (step 602). For example, the source electronic device may determine that information should be shared with the recipient electronic device in response to a decision that an event has occurred in which a user or recipient electronic device is involved (e.g., completion of a settlement transaction, mention of a product, etc.). Specifically, the source electronic device may implement a machine learning (ML) algorithm trained to detect utterances of brand names, product names, or personal names included in a given list. This detection algorithm may produce an output indicating that one of the brand names, product names, or personal names was uttered or displayed in the first stream of encoded data. In such embodiments, a source electronic device may determine, in response to a detection algorithm's determination that a given brand, product, or person has been spoken or displayed in a first stream of data in which that brand, product, or person is encoded, that information about that brand, product, or person should be transferred to a receiver electronic device.

[0096] Next, the source electronic device can generate or acquire a second stream of encoded data representing the information to be shared (step 603). For example, suppose the information to be shared relates to a settlement transaction completed using the source electronic device. In this scenario, the information related to the settlement transaction could be extracted by the signal manipulation platform, for example, by taking numerical values ​​directly from the source electronic device's processor, encoding them according to an appropriate encoding scheme, and then superimposing the calculated modulation onto the first stream of encoded data representing the original interface signal. Further information regarding encoding schemes may be provided below. However, it should be noted that, in general, the encoding scheme used is not important as long as the receiver electronic device "knows" what it should be looking for.

[0097] Next, the source electronic device may transfer information to the receiver electronic device by combining the second stream of encoded data with the first stream of encoded data to create a third stream of encoded data (step 604). More specifically, the source electronic device may superimpose the second stream of encoded data onto a portion of the first stream of encoded data. This third stream of encoded data may then be output for detection by the receiver electronic device (step 605).

[0098] From the perspective of a receiving electronic device, the presence or possibility of a broadcast may be detected directly or indirectly inferred based on its current location, its current use, or input provided by its user. For example, a receiving electronic device may activate an audio input component in response to a determination that it is located within a certain proximity to a given location. An example of an audio input component is a microphone. As another example, a receiving electronic device may activate a video input component in response to receiving input from its user indicating a request to do so. An example of a video input component is an image sensor that can generate image data in the form of a digital image, for example. The image sensor may be part of the camera module (or simply "camera") of the receiving electronic device.

[0099] Generally, the encoding and decoding schemes used by source and receiver electronic devices, respectively, are implementation-specific and therefore not described in detail. However, as an example, the IEEE (Institute of Electrical and Electronics Engineers) 802.15 series standards define coding, decoding, modulation, media access, and physical layer schemes for personal area networks. Any of these standards may be used to implement the millimeter-wave and OCC embodiments described above. In embodiments that rely on acoustic data transmission, one available protocol is Beeping, a communication protocol designed for transmitting data between electronic devices via ultrasound. It is also possible to develop custom standards to implement these embodiments.

[0100] Figures 7A–D include schematic diagrams illustrating how the extraction algorithm, detection algorithm, modulation algorithm, and superposition algorithm can cooperate to implement the approach described herein. These algorithms may be equivalent to the extraction algorithm 402, detection algorithm 404, modulation algorithm 406, and superposition algorithm 410 in Figure 4. Each of these algorithms is described in more detail below.

[0101] Figure 7A includes a schematic diagram showing how the extraction algorithm 702 can process interface signals to identify a suitable opportunity to transfer information to a receiver electronic device. The interface signals can initially be provided to the extraction algorithm 702 as input. As shown in Figure 7A, the extraction algorithm 702 may detect the context through several different approaches.

[0102] In one approach (also known as "content-based detection"), the extraction algorithm 702 can examine the content of the interface signal (e.g., spoken audio or displayed images), extract semantic meaning (e.g., the meaning of numbers, letters, words, and phrases) from the content, and then determine whether the semantic meaning matches a relevant topic for data transfer. The semantic meaning may be extracted using machine learning approaches. The relevant topic, on the other hand, may be determined, for example, by comparing the semantic meaning with a list of brands, products, or affiliates. If the semantic meaning matches a relevant topic, the extraction algorithm 702 may output a context tag representing the underlying content. For example, the context tag may indicate a prominent number, letter, word, or phrase from which the semantic meaning was extracted. If the semantic meaning does not match a relevant topic, the extraction algorithm 702 can continue examining the content of the interface signal.

[0103] In other approaches (also known as "state-based detection"), the extraction algorithm 702 can extract machine states and then determine whether the machine states match any relevant states in a data transfer. An example of a relevant state is reaching a predetermined interface (e.g., a checkout interface). Another example of a relevant state is interacting with a source electronic device for at least a predetermined time. If the machine state matches a relevant state, the extraction algorithm 702 may output a context tag representing the machine state. For example, the context tag may contain or imply information about the machine state. If the machine state does not match a relevant state, the extraction algorithm can continue its investigation of machine states.

[0104] Figure 7B includes a schematic diagram showing how the detection algorithm 704 can examine the output generated by the extraction algorithm 702 to determine whether it is appropriate to begin the transfer of information from the source electronic device to the receiver electronic device. The context tags generated by the extraction algorithm 702 as output may first be provided to the detection algorithm 704 as input. The detection algorithm 704 can then examine the context tags. For example, the detection algorithm 704 may compare each context tag to a predetermined list of context tags to determine whether there is a match. The predetermined list may include context tags for which relevant information, such as advertisements or promotional offers, is currently available. If the context tag matches an entry in the predetermined list, the detection algorithm 704 may output the matching context tag. By outputting the matching context tag, the detection algorithm 704 can begin the data transfer process, as will be described further later. Conversely, if the context tag does not match any entry in the predetermined list, the detection algorithm 704 may discard the context tag and begin the matching process with a new context tag.

[0105] Figure 7C includes a schematic diagram showing how the modulation algorithm 706 can modulate an interface signal so as to superimpose relevant information onto it. For illustrative purposes, the interface signal may be referred to as the “first signal.” Context tags generated by the detection algorithm 704 as an output may initially be provided to the modulation algorithm 706 as input. For each context tag, the modulation algorithm 706 can determine whether a second signal representing the data to be transferred for that context tag exists in the signal database. Each signal in the signal database may represent one or more data packets that convey information about the corresponding context tag. The signal database may be signal database 216 in Figure 2, signal database 312 in Figure 3, or signal database 408 in Figure 4. If a second signal exists in the signal database, the modulation algorithm 706 can retrieve or extract the second signal from the signal database. However, if a second signal does not exist in the signal database, the modulation algorithm 706 may generate a second signal representing the data to be transferred.

[0106] Next, the modulation algorithm 706 can determine the nature of the first signal. For example, based on an analysis of its content, metadata, or format, the modulation algorithm 706 may determine whether the first signal is an audio signal or a video signal. If the first signal is an audio signal, the modulation algorithm 706 can generate an appropriate audio modulation to encode the content of the second signal corresponding to the context tag. The audio modulation may be generated based on an acoustic data transmission scheme, as described above. If the first signal is a video signal, the modulation algorithm 706 can generate an appropriate video modulation to encode the content of the second signal corresponding to the context tag. The video modulation may be generated according to an existing modulation scheme (e.g., IEEE 802.15) or a proprietary modulation scheme. In some embodiments, these video modulations are replaced by the modulation algorithm 706 in the millimeter-wave domain so that these video modulations are detectable by a millimeter-wave radar system. As an output, the modulation algorithm can generate a second signal containing the data to be transmitted, modulated according to an appropriate modulation scheme.

[0107] Figure 7D includes a schematic diagram showing how the superposition algorithm 708 can superimpose the second signal output by the modulation algorithm 706 onto the first signal broadcast by the source electronic device. As input, the superposition algorithm 708 can take in (i) the first signal broadcast by the source electronic device and (ii) the second signal containing information to be transmitted to the receiver electronic device. As described above with reference to Figure 7C, the second signal can be modulated according to an appropriate modulation scheme based on the properties of the first signal.

[0108] Initially, the superposition algorithm 708 can extract device properties from the database. These device properties may relate to the source electronic device, the receiver electronic device, or both. By examining the device properties, the superposition algorithm 708 may determine or infer whether the receiver electronic device can detect the audio or video signal broadcast by the source electronic device. Furthermore, the superposition algorithm 708 may extract usage context properties from the database. Examples of usage context properties include the typical distance between the source electronic device and the receiver electronic device, and the typical duration of time the receiver electronic device spent in proximity to the source electronic device. In some embodiments, the database is maintained on the source electronic device, while in other embodiments, the database is maintained on another electronic device (e.g., a computer server) to which the source electronic device is communicably connected.

[0109] The superposition algorithm 708 can then combine the first and second signals to form a third signal. In some embodiments, this third signal is further manipulated and is therefore often referred to as the “prototype third signal”. After forming the prototype third signal, the superposition algorithm 708 may estimate the detectability of the second signal based on equipment properties and usage context properties extracted from a database. Detectability may depend on (i) parameters of the source electronic device, such as display size and audio volume; (ii) parameters of the receiver electronic device; and (iii) environmental factors, such as distance and background noise. If it is determined that the detectability of the second signal is low, the superposition algorithm may modulate the first and second signals to increase detectability. Furthermore, the superposition algorithm 708 may determine, taking into account equipment properties and usage context properties, the degree to which the second signal is perceptibly perceived within the prototype third signal. If it is determined that it is likely to be perceptibly perceived, the superposition algorithm 708 may modulate the second signal to reduce its noticeability. The prominence of the second signal can be reduced, for example, by reducing the presence or magnitude of frequencies within the audible range, by reducing the contrast difference of the visual content, or by increasing the flicker rate beyond the visible range. As shown in Figure 7D, the superposition algorithm 708 can then determine whether the prototype third signal still meets the detectability requirements. If the prototype third signal still meets the detectability requirements, the superposition algorithm 708 can output the third signal in a finalized form in preparation for broadcast by the source electronic device. This third signal may not only include data about the corresponding context tag, but may also be presented alongside the first signal in a manner that is detectable by the receiver electronic device but minimally detectable (or not detectable at all) by the user of the source electronic device. If the prototype third signal does not meet the detectability requirements, the superposition algorithm 708 can continue the modulation process as shown in Figure 7D. [Example Usage]

[0110] Figures 8-9 illustrate two different scenarios in which the approaches described herein may be implemented to transfer relevant information to an individual (more specifically, their electronic device) through the modulation of an audible or visual signal.

[0111] Figure 8 illustrates how links can be intelligently embedded in a speech signal based on an analysis of the speech contained within the speech signal. One possible implementation of the approach described herein involves creating a speech embedding within the speech that functions as a “payment link.” By recognizing relevant words (e.g., “amount,” “dollars,” etc.) and phrases (e.g., “transaction complete,” “payment received,” etc.) and then broadcasting payment-related information associated with those “cures” as metadata for each relevant word, payment-related information can be embedded in the user’s speech, sometimes in real time. This makes it possible to generate advertisements within an individual’s utterance in near real time by recognizing phrases that constitute a marketing or advertising opportunity.

[0112] As shown in Figure 8, system 800 may include interface signals (here, a live broadcast recording 802, a database 804 containing information about marketing opportunities such as advertisements, and a detection algorithm 806). In this scenario, the detection algorithm 806 may be responsible for transcribing, establishing, or understanding the content of the live broadcast recording 802. Thus, the detection algorithm 806 may be called a “transcription algorithm”. Next, the detection algorithm 806 may consider the content of the live broadcast recording to determine whether any marketing opportunity contained in the database is appropriate or suitable. In other words, the detection algorithm 806 can determine whether the content “fits” any marketing opportunity contained in the database. Fit may be determined based on, for example, the characteristics or meaning of words, user preferences, attributes of potential audiences, etc. If it fits, the appropriate information may be transferred to the broadcast system responsible for the output of the live broadcast recording 802.

[0113] To help illustrate the adaptability of the approach described herein, several additional scenario examples are provided below. As one example scenario, suppose an influencer is currently engaged in a live broadcast. In such a scenario, system 800 can detect the product the influencer is talking about. Once a “target word” is detected by system 800, system 800 can superimpose an ultrasonic data packet containing information about the product onto the next word or phrase. Each time someone records the live broadcast, that person can obtain the relevant payment details. Furthermore, each time a link is opened or a product is purchased via the link, the influencer may receive a reward. As another example, in a podcast, system 800 can superimpose various advertisements onto words or phrases that match marketing opportunities in categories selected by the user. In previously recorded podcasts, updated marketing opportunities may be used to embed newer advertisements into the live broadcast.

[0114] Figure 9 illustrates how the approach described herein can be used to facilitate the secure transfer of sensitive information. More specifically, Figure 9 shows how information related to (e.g., required for) a settlement transaction can be communicated visually through the modulation of visual signals. An interesting application of the approach described herein includes digital receipts. However, the approach described herein can also be used to facilitate settlement beacons, special offers and promotions, advertising, and the like.

[0115] For example, suppose an individual completes a payment transaction using a legacy system such as an advertising or payment system. This legacy system can seamlessly communicate payment-related but not overly sensitive data to the recipient's electronic device. This can be achieved by embedding visual codes in its display. These visual codes can be investigated, analyzed, or detected by sensors (e.g., a millimeter-wave radar system) built into the recipient's electronic device. The received information can then be used by another computer program running on the recipient's electronic device to provide functions such as digital receipts, product information, advertisements, and special offers.

[0116] These visual codes can be embedded in the underlying video signal in the form of interlaced lines whose frequencies encode data, but they may be too fast to be noticed by the viewer of the display content. Overall, this approach has low requirements for legacy systems. Therefore, the approach described herein can be implemented simply by modifying the firmware or software that manages the display of the content.

[0117] It should be noted that the receiving electronic device can identify the moment when these visual codes may be present (for example, based on location) and then switch on their detection. In embodiments where detection is achieved using a millimeter-wave radar system, the user may not need to remove the receiving electronic device from their pocket, bag, etc. Instead, the millimeter-wave radar system may be able to detect these visual codes through clothing. UI functions may also be managed based on the present visual codes. For example, UI functions may be enabled through the angular dependence of the code signals and the alignment of the receiving electronic device and the source electronic device.

[0118] As shown in Figure 9, source electronic devices 902a-b can initially display one or more digital images. An algorithm implemented either internally or externally can generate, for example, a “flash code” that carries information suitable for the content of these digital images. When receiver electronic devices 904a-d detect a situation in which a flash code may be present (for example, based on the user being located near the source electronic device), they switch on their millimeter-wave radar system. Alternatively, the receiver electronic device may actively scan nearby visual codes at a predetermined frequency. The data represented by the flash code can be read by the receiver electronic device via the millimeter-wave radar system. Subsequently, the individual associated with the receiver electronic device is presented with the results of the data capture (for example, a digital receipt or advertisement).

[0119] Figure 9 illustrates two different use cases. The first use case relates to a special offer via an advertising interface (also called an "advertising screen"). For example, suppose an individual passes an advertising screen 902a during a routine commute on public transport (e.g., subway or bus). This advertising screen 902a may represent a source electronic device. A recipient electronic device 904a associated with the individual may actively scan the surrounding environment every few seconds to identify the presence of a visual code. On public transport, the content of the source electronic device 902a is modulated so that a flash code related to the displayed advertisement can be embedded in the video signal. The advertisement may prompt nearby individuals to check their respective recipient electronic devices 904a-c for a surprise discount or promotion on a product offered. As mentioned above, a sensor such as a millimeter-wave radar sensor can detect the modulation of the video signal without being removed from trousers, a jacket, etc. Thus, without needing to be removed from a pocket, the recipient electronic device 904a can detect information from the flash code and store the information for subsequent use (e.g., if the information relates to an offer or promotion). Subsequently, the individual may check the computer program (e.g., a payment application) running on the recipient's electronic device to determine whether the information is still usable.

[0120] The second use case relates to digital receipts for legacy payment systems. For example, suppose an individual with access to mobile phone 904d makes a purchase at a store. And suppose a computer program running on mobile phone 904d, which is authorized to make digital payments, has its reading function enabled. This computer program may be called a “payment application.” In such a scenario, when the payment application determines that the individual is in proximity to the payment system, it may initiate a detection function supported by the millimeter-wave radar sensor contained in mobile phone 904d. When the individual’s transaction is processed by the payment system, the modified firmware may embed a code in it that represents the purchased items, the transaction amount, and a timestamp. The individual may complete the transaction using a payment card (e.g., a credit or debit card) programmatically associated with the payment application. The visual code output by source electronic device 902b can be read by receiver electronic device 904b. Furthermore, the payment application may be able to match the settlement of those expenditures to the item list and timecode. This is true even if the receiver electronic device detects multiple different data packets while in proximity to the payment system. Therefore, individuals may be provided with digital receipts. [Processing System]

[0121] Figure 10 is a block diagram showing an example of a processing system 1000 capable of implementing at least some of the operations described herein. For example, the components of the processing system 1000 may be hosted on an electronic device including a video output component or an audio output component. As another example, the components of the processing system 1000 may be hosted on an electronic device including an operating platform capable of manipulating interface signals that are audibly or visually detectable by another electronic device for the purpose of information transfer.

[0122] The processing system 1000 may include a processor 1002, main memory 1006, non-volatile memory 1010, a network adapter 1012 (e.g., a network interface), a video display 1018, an input / output device 1020, a control device 1022 (e.g., mechanical input such as a keyboard, pointing device, or buttons), a drive unit 1024 including a recording medium 1026, or a signal generator 1030, all of which are communicatively connected to the bus 1016. The bus 1016 is illustrated as an abstraction representing one or more physical buses and / or point-to-point connections connected by appropriate bridges, adapters, or controllers. Thus, the bus 1216 may include a system bus, a PCI (Peripheral Component Interconnect) bus, a PCI-Express bus, a HyperTransport bus, an ISA (Industry Standard Architecture) bus, a SCSI (Small Computer System Interface) bus, a USB (Universal Serial Bus), and an I 2 This may include a C (Inter Integrated Circuit) bus or a bus conforming to IEEE standard 1394.

[0123] The processing system 1000 may share a computer processor architecture similar to that of a computer server, router, desktop computer, tablet computer, mobile phone, video game console, wearable electronic device (e.g., watch or fitness tracker), network-connected ("smart") device (e.g., television or home assistant device), augmented or virtual reality system (e.g., head-mounted display), or another electronic device capable of executing (sequentially or otherwise) an instruction set that specifies the action(s) to be performed by the processing system 1000.

[0124] Although the main memory 1006, the non-volatile memory 1010, and the recording medium 1024 are shown as a single medium, the terms “recording medium” and “machine-readable medium” should be interpreted to include a single medium or multiple mediums that store one or more sets of instructions 1026. Furthermore, the terms “recording medium” and “machine-readable medium” should be interpreted to include any medium capable of storing, encoding, or carrying a set of instructions for execution by the processing system 1000.

[0125] Generally, routines performed to implement embodiments of the present disclosure may be implemented as part of an operating system or a particular application, component, program, object, module, or sequence of instructions (collectively referred to as a "computer program"). A computer program typically comprises one or more instructions (e.g., instructions 1004, 1008, 1028) set at various times in various memories and storage devices within a computing device. When read and executed by processor 1002, the instructions cause processing system 1000 to perform actions to implement various embodiments of the present disclosure.

[0126] While embodiments have been described in the context of fully functional computing devices, those skilled in the art will understand that various embodiments can be distributed as various forms of program products. This disclosure applies regardless of the specific type of machine- or computer-readable medium used to actually achieve distribution. Further examples of machine- and computer-readable media include recordable type media such as volatile memory devices, non-volatile memory devices 1010, removable disks, hard disk drives, optical discs (e.g., Compact Disk Read-Only Memory (CD-ROM) and Digital Multipurpose Disc (DVD)), and transmission type media such as cloud-based storage and digital and analog communication links.

[0127] The network adapter 1012 enables the processing system 1000 to mediate data between entities outside the processing system 1000 and the network 1014 through any communication protocol supported by the processing system 1000 and the external entities. The network adapter 1012 may include a network adapter card, a wireless network interface card, a switch, a protocol converter, a gateway, a bridge, a hub, a receiver, a repeater, or a transceiver including a chip (for example, enabling communication via Bluetooth, Wi-Fi, or NFC). [remarks]

[0128] The foregoing description of various embodiments of the claimed subject matter is provided for illustrative and explanatory purposes only. It is not intended to be exhaustive or to limit the claimed subject matter to the exact form disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments have been selected and described to best illustrate the principles of the invention and its practical applications, thereby enabling those skilled in the art to understand the claimed subject matter, various embodiments, and various modifications suitable for the particular intended use.

[0129] While the detailed description illustrates specific embodiments and the intended best mode, the Art can be implemented in many ways, no matter how detailed the description may seem. Embodiments, while encompassed herein, may differ considerably in their implementation details. Any specific term used to describe particular features or aspects of various embodiments should not be construed as meaning that the term is redefined herein to limit the Art to any particular characteristic, feature, or aspect of the Art to which it relates. In general, the terms used in the following claims should not be construed as limiting the Art to any specific embodiment disclosed herein unless those terms are expressly defined herein. Therefore, the actual scope of the Art encompasses not only the disclosed embodiments but also all equivalent methods of carrying out or implementing those embodiments.

[0130] The terminology used herein has been selected primarily for readability and explanatory purposes. It has not been selected to define or enclose the subject matter. Accordingly, the scope of this art is intended to be limited not by this detailed description, but rather by any claims issued in an application relating thereto. Accordingly, the disclosure of various embodiments is intended to illustrate, but not to limit, the scope of the art set forth in the following claims. [Industrial applicability]

[0131] This disclosure is applicable to wireless communication of information.

Claims

1. The first electronic device outputs a first signal that is visually perceptible to the user and represents a video signal displayed by the video output component of the first electronic device. The first electronic device receives an input indicating an instruction to communicate information about the content of the first signal to a second electronic device located in close proximity to the first electronic device. The first electronic device superimposes a second signal onto the first signal, which is not visually perceptible to the user but indicates information, so as to generate a third signal, wherein the superposition introduces visual modulation to the video signal over a time interval, and the visual modulation affects only a portion of each frame of the video signal included in the time interval, and The first electronic device outputs the third signal for detection by the second electronic device, so that the second electronic device can detect the second signal even though the second signal is not visually perceptible to the user. A method for providing this.

2. The method according to claim 1, wherein the second signal is superimposed on the first signal in such a manner that the modulation of the first signal is not visually perceptible to the user.

3. The aforementioned superposition is, Based on an analysis of the content of the first signal, the time interval in which the second signal is superimposed is determined, The first signal and the second signal are combined such that the second signal matches the time interval, The method according to claim 1, comprising:

4. The method according to claim 1, wherein (i) the content of the second signal, or (ii) a method by which the second signal is superimposed on the first signal, uniquely associates the second signal with the second electronic device.

5. When executed by the processor of an electronic device, the electronic device: To output a first signal that is visually detectable by the user and represents a video signal displayed by the video output component of the electronic device, Receiving input that indicates instructions to transmit information to other electronic devices, Superimposing a second signal onto the first signal that is not visually detectable by the user and contains information, thereby generating a third signal, wherein the superposition introduces visual modulation to the video signal over a time interval, and the visual modulation affects only a portion of each frame of the video signal included in the time interval, and Outputting the third signal for detection by the other electronic device so that the other electronic device can detect the second signal, even though the second signal is not visually detectable by the user. A non-temporary medium in which instructions for performing an action comprising the above are stored.

6. The aforementioned operation is, The non-temporary medium according to claim 5, further comprising identifying a portion of the first signal superimposed with the second signal by examining the content of the first signal in real time.

7. The aforementioned operation is, Based on an analysis of the content of the first signal, the time interval for superimposing the second signal is determined, and The non-temporary medium according to claim 5, further comprising combining the first signal and the second signal such that the second signal matches the time interval.

8. The aforementioned operation is, The further comprising specifying a modulation scheme for communicating the information to the other electronic device based on the characteristics of the electronic device, The superposition is performed according to the modulation scheme, in the non-temporary medium according to claim 5.

9. The non-transient medium according to claim 8, wherein the characteristics are display size, display type, model, or intended application.

10. The modulation scheme specifies at least one characteristic of the second signal, the non-transient medium according to claim 8.

11. The non-transient medium according to claim 5, wherein the second signal is one of a plurality of signals superimposed on the first signal and is therefore included in the third signal.

12. The non-transient medium according to claim 11, wherein each of the plurality of signals corresponds to a different time interval at which the third signal is output for detection by the other electronic device.

Images