Device positioning

Abstract

The present invention relates to a method and associated system for connecting a primary device to an external device, where the primary device is configured to receive acoustic signals within a predetermined frequency range and measure the location of the acoustic transmitter, and the secondary device includes the acoustic transmitter, the method comprising: a) the primary device detecting the presence of an external secondary device; b) the primary device requesting the secondary device to transmit an acoustic signal; c) the secondary device transmitting the predetermined acoustic signal; d) the primary device receiving the acoustic signal; and e) the primary device calculating the location of the secondary device based on the received acoustic signal.

Description

[Technical Field]

[0001] The present invention is based on calculating the relative position of one or more secondary devices with respect to a primary device. [Background technology]

[0002] For example, it is well known to connect different types of electronic devices to extend a desktop screen. Typically, this process is partially manual, where a user may select and position the external screen relative to the main computer through a computer interface. This requires knowledge of the system and is time-consuming whenever a new setup is required.

[0003] The present invention aims to simplify the setup process for combining external devices in a system to provide a unified experience to the user. In WO2011 / 042748 a system is proposed in which two or more screens interact such that the relative positions and relative orientations (e.g. distances, angles, etc.) between them are monitored using acoustic measurements. This gives the possibility to use directional gestures or pointer movements to move objects from one screen to another even though their relative positions may change.

[0004] EP3968295 describes a communication system and method for electronic devices, including two or more devices, that aims to detect whether another device is within a certain distance. This process begins, for example, using the Bluetooth communication protocol, which may be followed by ultrasonic communication to measure the distance. Each device contains one microphone and one speaker, both of which must be active to determine the distance, and relative orientation is considered problematic as it reduces the accuracy of the distance measurement. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2011 / 042748 [Patent Document 2] European Patent Application Publication No. 3968295 [Patent Document 3] International Publication No. 2022 / 189140 Summary of the Invention [Problem to be solved by the invention]

[0006] It is an object of the present invention to provide an improved system or method for calculating and preferably monitoring the relative position and preferably orientation of one or more secondary devices with respect to a primary device. This is achieved as disclosed in the accompanying claims. [Means for solving the problem]

[0007] The present invention is therefore based on communication between a primary device and at least one secondary device using acoustic signals transmitted from one or more acoustic transducers in the secondary device, received by one or more transceivers in the processing device of the primary device, and processed to determine the relative position of the secondary device with respect to the primary device. This processing can be performed in specialized or general-purpose processors, including inference engines for deep neural networks (DNNs). The acoustic signals are either stored in a playout buffer in the secondary device or generated on-the-fly by either the primary or secondary device. If the secondary device does not have resources, including processing power, to generate the acoustic signals, the acoustic output signal can be generated by the primary device. Note that the primary device must be able to transmit the acoustic signals using the acoustic transducers in the secondary device through a viable communication channel, such as a wired or wireless connection. The wired connection may go through a forwarding device capable of handling multiple secondary devices (e.g., wired or wireless docking stations).

[0008] The primary device may initiate the process upon receiving an acoustic or Wi-Fi signal, for example when the second device has a repeating beacon or acts on a request from the user to connect, or when it detects active use by the user, for example a computer mouse registering activity or pressure or a "click" from an inertial measurement unit (IMU), although the primary device may also be activated when it detects a change in sound level or a recognizable sound.

[0009] In one embodiment, the computer mouse may therefore provide a signal to activate distance and orientation measurements, and if the distance is too large or the position is not suitable, for example if a right-handed mouse is registered on the left side of the primary device, the connection may be rejected, possibly simultaneously accepting the connection when a second primary device registers the position of the mouse.

[0010] An advantage of using acoustic signals to initiate connection, distance, or orientation measurements is that, as discussed in WO 2022 / 189140 (P6000), acoustic signals are generally confined to a relatively small volume, particularly within a sealed chamber, limiting unnecessary system initiation. Transmission of signals containing access information, such as password information or IP addresses, is also physically confined to the system, reducing noise and activity registered by other nearby devices using electromagnetic communication. The acoustic signals transmitted by one or more transducers in the secondary device may be either audible or inaudible (e.g., ultrasonic, infrasonic, etc.) or a mixture of the two signal types. If the positioning process using the first signal type is difficult or inconclusive, it is possible to first use one signal type (e.g., inaudible) and later use the other (e.g., audible). If the secondary device includes multiple transducers, different acoustic signals may be transmitted in parallel on different transducers.

[0011] The invention is further based on the concept of using recognizable audible or inaudible sounds (e.g. infrasonic, ultrasonic) that can be distinguished from other acoustic sources in the environment, including the output from any device (e.g. radio, TV, smart speaker, Hi-Fi system, smartphone, etc.) or any object that produces sound (e.g. people, pets, coffee grinder, PC fan, dishwasher, etc.) that explicitly plays out sound. In this way, power consumption can be limited by reducing acoustic transmissions when there is a certain risk of connection failure.

[0012] The invention will be described in more detail below with reference to the drawings, which illustrate the invention by way of example. [Brief explanation of the drawings]

[0013] [Figure 1a] FIG. 1 illustrates an embodiment of the present invention having a primary device and at least one secondary device. [Figure 1b]FIG. 1 illustrates an embodiment of the present invention having a primary device and at least one secondary device. [Figure 1c] FIG. 1 illustrates an embodiment of the present invention having a primary device and at least one secondary device. [Figure 1d] FIG. 1 illustrates an embodiment of the present invention having a primary device and at least one secondary device. [Figure 2] FIG. 1 is a diagram showing a processing sequence according to the present invention. [Figure 3] FIG. 10 is a diagram illustrating an alternative sequence. DETAILED DESCRIPTION OF THE INVENTION

[0014] As shown in Figures 1a to 1d, a system according to the present invention comprises a primary device 1 and at least one secondary device 2 capable of wireless or wired communication. The primary device 1 includes at least one first acoustic transducer 4 capable of receiving acoustic signals within a predetermined frequency range, and the secondary device includes at least one second acoustic transducer 3 capable of transmitting signals within the predetermined frequency range.

[0015] According to a preferred embodiment of the invention, the devices each have two transducers with a known distance between them. The system is therefore able to measure the orientation of a second device relative to a first device in a manner known per se based on the time of flight (ToF) or phase difference for acoustic signals from the two transmitters to the two receivers. However, other solutions may also be possible, for example by transmitting a structured acoustic beam front that indicates the direction of the transmitter.

[0016] In Figure 1a, two devices 1, 2 are connected through a wired connection 5. When a connection is detected, the first device 1 is configured to send a signal to the second device 2 instructing the second device 2 to transmit one or more acoustic signals from a transmitter 3. The signals are then received by the first device 1 through a receiver 4.

[0017] 1b-1d, a first device may be able to communicate with a second device using Wi-Fi or other wireless communication means. The first device typically waits until it detects a second device in the vicinity and then transmits a signal instructing the second device to transmit an acoustic signal used to determine the relative position between the devices. This may be repeated until another second device arrives, thus providing, for example, a multi-screen environment or using several audio sources, where only the new second device is asked to transmit the signal required to determine its location.

[0018] The configuration in Figure 1b may consist, for example, of a stereo system with two speakers controlled by the first device, or a laptop with two external monitors. Figures 1c and 1d show other possible configurations.

[0019] An audio signal may not be usable if audible noise in the environment or from concurrent audio played out on the secondary device makes the audio signal undetectable by the primary device. Similarly, an audio signal may be undetectable if interference from inaudible noise in the environment or intermodulation effects from playing concurrent audio signals on acoustic transducers in the secondary device prevent correct positioning. Because these noise sources may not be present all the time or may potentially be periodic or random in nature, it may be possible for the primary device to transmit audio signals during periods when the audible or inaudible noise sources discussed above are manageable or absent. With loopback access to the concurrent audio output stream, the positioning process can predict periods in the audio signal that would be usable for positioning using the audio signal. This prediction may also be made using a deep neural network inference engine trained on audio data or predictions based on signal processing of the outgoing audio signal. The amplitude of the audio signal may be increased to improve the SNR to allow the positioning process to coexist with other noise sources. There is also the option of changing the acoustic signal to one with a better SNR (eg, changing the signal type, frequency or frequency band, etc.).

[0020] More specifically, FIG. 2 illustrates a process in accordance with the present invention for connecting and positioning a secondary device. 10: The primary device detects a secondary device, such as an external monitor, either by being physically connected to the device or by being connected using a wireless protocol. 11: Once connected, the positioning process begins. 12: The system, preferably the primary device, listens for noise in the environment. 13: The secondary device is instructed to transmit an acoustic signal with predetermined characteristics. If noise is detected, the acoustic signal may be chosen to be in a different frequency range or between noise signals if the noise is periodic. 14: The primary device receives the acoustic signal. 15: Based on the signals received at the acoustic receiver, the relative position is calculated. 16: If a relative position is found, the process is stopped.

[0021] If a new secondary device is detected, the process may be repeated.

[0022] 3 illustrates a process such as that described above in which the relative positions between devices are dynamic and changing, or additional secondary devices appear or are detected. In that scenario, the system may repeat the positioning process at a selected rate, with step 17 including returning to step 12 and listening for noise.

[0023] As shown in Figures 1b-1d, there may be multiple secondary devices. If so, the secondary devices are located by the primary device in parallel or one after the other. The order of location is controlled by the primary device or by protocol requests from the secondary devices using the appropriate service protocol. The secondary device or the primary device, if controlling the acoustic output signal, may transmit device-specific acoustic signals if location is performed in parallel to keep devices apart in the location process.

[0024] In a preferred embodiment of the present invention, the primary device is a laptop, e.g., part of a videoconferencing system, and the secondary device is an external monitor equipped with one or more speakers. The monitor is connected to the laptop using a cable (e.g., HDMI, USB-C, etc.) or wireless technology (e.g., Wi-Fi, Bluetooth, etc.) or optical communication technology, but generally does not include processing capabilities for generating output signals or including stored acoustic signals in a playout buffer. Therefore, signals must be played out from a playout buffer or generated on the fly by the primary device and sent over acoustic transducers in the secondary device using a cable or wireless connection (i.e., HDMI, USB-C, etc.) between the primary and secondary devices. In this embodiment, relevant information about the audio system on the external monitor is available in Extended Display Identification Data (EDID), which is embedded in these external monitors and read by the laptop when the displays are connected. EDID information may include the monitor size, speaker location, number of speakers, sampling rate, etc. All this information can be included in a positioning deep neural network (DNN) as parameters in the signal processing of the positioning process or as processed input features. Furthermore, in a videoconferencing system, for example, the distance and orientation of devices can be used for sound and video representation in a room, for example, by adapting the system to the relative positions of the units.

[0025] The positioning process may be configured to begin when the laptop detects that there is an external monitor with one or more speakers connected to the laptop and available. The laptop may discover the external monitor when a device manager in the device detects that a new secondary device with at least one speaker is accessible to the primary device through an existing connection, such as HDMI, Wi-Fi, Bluetooth, or USB-C. Once the primary and secondary devices are logically connected, the device manager reports the availability of speakers on the monitor. The discovery process may be triggered when the device boots, resumes from sleep mode, or restarts the process of discovering secondary devices. The device polling or notification process may occur in a driver or application or a background service in the device.

[0026] If an external monitor with speakers is detected, the positioning process begins playing out an acoustic signal on one or more of the external monitor's speakers. If the secondary device is not within range of the primary device and the positioning process fails, no location information is known and the default behavior of the primary device is respected. This can occur if the primary and secondary devices in question are located far enough apart or in different rooms or spaces that positioning is feasible, which severely limits at least the ultrasonic signal. If the acoustic signal is successfully received by the laptop transducer, the positioning process determines the relative position of the external monitor to the laptop. Once the relative position is known (e.g., left, right, top, bottom, front, back, top-bottom, top-top, left-left, left-right, etc.), the laptop can optimize the setup and positioning of all available displays, either automatically or by prompting the user, using a platform-specific API or a user-friendly UI to complete the display setup. This setup is commonly known as display extension, and all available displays can be used by the laptop user.

[0027] In another embodiment, the laptop is connected to two or more external monitors, and the positioning process positions one external monitor after the other, or in parallel.

[0028] In another embodiment, the laptop is connected to a sound bar or external speakers and the positioning process positions them.

[0029] In another embodiment, the laptop may use the screen of another device, such as a laptop, a standalone computer, a TV, a video conferencing board, a tablet, a smartphone, etc., as well as the normal use of any external monitor. Generally, these secondary devices may include their own processing capabilities, allowing them to transmit acoustic signals from their own speakers. However, control of when the secondary devices transmit acoustic signals should preferably be left to the primary device to ensure that the positioning processes of all available secondary devices are performed properly, whether in parallel or sequentially. These devices are generally not connected by cables but rely on a wireless connection (e.g., Wi-Fi, Bluetooth, ultrasonic, etc.) between the primary and secondary devices. The primary and secondary devices can use any type of discovery protocol to detect each other via the wireless connection. Securing the discovery protocol may be necessary to ensure that only secondary devices accessible to the user are detected. A useful alternative method for detecting secondary devices is to have the secondary device potentially controlled by the primary device and transmit an inaudible ultrasonic signal to ensure that both secondary devices are in the same space and acoustically within range of the primary device. The primary device sends commands or complete acoustic signals to the secondary device to enable the secondary device to transmit the correct acoustic signals from at least one of its acoustic transducers.

[0030] In one embodiment, the laptop plays out an acoustic signal adapted to the current acoustic noise in the device's environment to improve the signal SNR based on an initial acoustic noise measurement of the acoustic signal received by the at least one acoustic transducer.

[0031] In one embodiment, the laptop is connected to two or more external monitors, and the positioning process positions all of the external monitors simultaneously by playing out a unique acoustic signal on all of the external monitors simultaneously.

[0032] In one embodiment, the positioning process uses either changes in static acoustic echoes from the environment or sensor events from other sensors (e.g., hinge sensors, IR sensors, ToF sensors, IMUs, cameras, etc.) that indicate the device in question is being moved to detect whether either the primary or secondary device is currently being moved or undergoing a movement, such as a button or mouse press. The primary device may restart the positioning process if either the primary or secondary device appears to be moving. If possible, the secondary device may send an acoustic signal to inform the primary device that the secondary device has moved if a motion sensor is embedded within the secondary device. The secondary device may delay sending the acoustic signal until the device is no longer moving. In that case, the primary device may wait until the moving device is no longer moving, and the positioning process may be restarted to ensure that the positioning information is correctly updated.

[0033] According to one embodiment, one or more physical or virtual sensors (e.g., an IMU) may be used to detect movement of either the primary or secondary device, triggering the positioning or orientation measurement process to run again. IMU sensors typically have low power consumption, which may be advantageous in portable devices. If the IMU sensor in the primary device indicates device movement during the positioning process, the positioning process should be aborted by the primary device. If the secondary device detects movement of the IMU sensor, it should send a predefined signal, if possible, that the positioning process failed due to device movement. Other criteria or conditions that must be satisfied before a connection is set up include device state, device configuration (hinge angle, folded, unfolded, screen disconnected, plugged, etc.), device orientation, power state, and biometric security, such as, but not limited to, face ID, voice recognition, and fingerprint. Gesture, i.e., how either or both devices are handled (e.g., shaken, lifted, swung, etc.), may be yet another condition that must be satisfied for a connection to be initiated. Furthermore, the positioning itself may provide the basis for providing full communication between devices, for example by requiring the second device to be in front of the screen and / or within a selected distance range in order to connect to it.

[0034] In one embodiment, the positioning process may be run at regular intervals to detect when at least one secondary device has moved relative to the primary device, the intervals may be user configurable.

[0035] In another embodiment of the invention, user input can be used to indicate whether the user wants to use a dedicated external monitor. This can be accomplished by blocking mouse movement on a laptop, ensuring that when the user moves the mouse from the current screen to the screen on the right, the right screen is automatically expanded to the right and turned on or dimmed. This can allow the user to configure power savings and reduce power consumption by keeping only the screen that is actively being used on at all times. Other screens can be turned off or dimmed for power saving reasons.

[0036] In another embodiment, when a secondary device is connected to the primary device, the primary device may play audio (e.g., podcasts, music, videos, etc.) that can be rerouted and played out on speakers of the secondary device. In this case, the output signal from the positioning process should be mixed with the current audio output and played out in parallel on the same speakers. The output signal may be an ultrasonic or audible signal that can be distinguished from the audio itself.

[0037] In another embodiment, when a secondary device is connected to the primary device, the primary device may play audio (e.g., podcasts, music, videos, etc.) that can be rerouted and played out on speakers on the secondary device. If the output signal from the positioning process cannot be mixed with the audio output and sent from speakers on the secondary device, the positioning process may use the audio output for its positioning process. This may require the positioning process to either gain access to the audio output or have a priori information about the audio itself to simplify the positioning process. If the positioning process does not have access to the audio signal being sent, the positioning process may have an increased failure rate.

[0038] The present invention may use existing speakers and microphones on the devices, such as one speaker on one device and two spatially separated microphones on the other device, where the time of flight from the speakers to the different microphones provides a measure of relative orientation. The reverse may also be possible, where two speakers on one device may transmit known signals, and the signals received by the microphones may be analyzed based on the known signals, for example, by finding the phase difference between the received signals.

[0039] The process may be initiated manually by a user, or by detected activity on one of the devices, such as movement that may indicate that distance or orientation may have changed, or by detecting a change in background sound levels that may be recognized as a person entering the room or a voice command, which may lead to the transmission of an activation signal, preferably acoustic, from one of the devices.

[0040] In summary, the present invention relates to a communication system for electronic devices, including a primary device and at least one secondary device. The devices include communication means configured to communicate with each other using a known communication protocol, such as a wired or wireless connection. The primary device is configured to detect the presence of secondary devices in the communication system according to the corresponding protocol. The secondary devices are also configured to transmit at least one acoustic signal within a frequency range, the primary device is configured to receive the acoustic signals within said frequency range, and the primary device is configured to determine the location of the secondary device based on the received signals, including the direction and / or distance to the secondary device.

[0041] The primary device preferably initiates the acoustic signal from the secondary device by sending a request signal, which can be an acoustic signal or sent through a communication means. Alternatively, the secondary device may be configured to send acoustic signals in a predetermined sequence until it receives a confirmation signal from the primary device acknowledging the communication, e.g., connection.

[0042] At least one of the primary and secondary devices may be configured to measure and analyze background noise using well-known methods and to transmit an acoustic signal within a detected low-noise period or frequency range.

[0043] The system may include more than one secondary device, in which case acoustic transmission may be initiated by the primary device individually to each of the secondary devices.

[0044] If the primary and / or secondary devices comprise two acoustic transducers separated by a known distance, the system may measure relative position using acoustic propagation time or phase and the distance between the transducers located on the secondary device. The devices may also include an inertial sensor or measurement unit for initiating relative position or location measurement upon detection of movement, and therefore a possible change in position, of one of the devices. Possibly, the process may be initiated when movement is constituted by pressure exerted on one of the devices, such as the click of a computer mouse or key.

[0045] The present invention also relates to a method for connecting a primary device to an external secondary device, both of which are provided with communication means for communicating between the devices, such as a cable or a wireless system, wherein the primary device is configured to receive acoustic signals within a predetermined frequency range using an acoustic transducer and to determine the location of an acoustic transmitter and a secondary device containing the acoustic transmitter.

[0046] The method is: a) a primary device detecting the presence of an external secondary device through a communication means; b) the primary device requesting, via a communication means, the secondary device to transmit an acoustic signal having specified characteristics; c) the secondary device transmitting a predetermined acoustic signal; d) the primary device receiving an acoustic signal; e) the primary device locating the secondary device based on the received acoustic signal.

[0047] The detection in step a) may be triggered by detecting an initial acoustic signal transmitted from the secondary device or through an established communication channel, e.g., a wired or Wi-Fi connection, possibly initiated by a registered human activity on the second device, such as a movement, key, touchpad or computer mouse press, etc. Steps b) to e) may be repeated in a predetermined sequence and / or upon the occurrence of measured movement of the secondary device. [Explanation of symbols]

[0048] 1 Primary device, first device 2 secondary device, second device 3. First acoustic transducer, transmitter 4 Second acoustic transducer, receiver 5 Wired Connection

Claims

1. 1. A communication system for electronic devices including a primary device and at least one secondary device, the devices configured to communicate with each other using a known communication protocol, the primary device configured to detect the presence of a secondary device, the secondary device configured to transmit at least one acoustic signal within a frequency range upon a request communicated from the primary device, the primary device configured to receive acoustic signals within the frequency range, and the primary device configured to determine the location of the secondary device based on the received acoustic signals.

2. The system of claim 1 , wherein the primary device is configured to initiate the acoustic signal from the secondary device.

3. The system of claim 1 , wherein the secondary device is configured to transmit the acoustic signals in a predetermined sequence until it receives a confirmation signal from the primary device acknowledging communication, e.g., connection.

4. 2. The system of claim 1, wherein at least one of the primary device and the secondary device is configured to measure background noise, and the system is configured to transmit the acoustic signal during a detected period of low noise.

5. The system of claim 1 , including at least two secondary devices, the transmission of the acoustic signal from each of which is initiated by the primary device.

6. 2. The system of claim 1, wherein at least one of the primary device and the secondary device comprises two transducers separated by a known distance, and the system is configured to measure relative position using acoustic transit time and the distance between the transducers located on the secondary device.

7. The system of claim 1 , wherein one of the primary device or the secondary device includes an inertial measurement unit configured to initiate measuring the location.

8. 1. A method for connecting a primary device to an external device, the primary device being configured to receive acoustic signals within a predetermined frequency range and determine the location of an acoustic transmitter, and a secondary device including the acoustic transmitter; a) the primary device detecting the presence of an external secondary device; b) the primary device requesting the secondary device to transmit an acoustic signal; c) the secondary device transmitting a predetermined acoustic signal; d) the primary device receiving the acoustic signal; e) the primary device calculating a location of the secondary device based on the received acoustic signal; A method comprising:

9. The method of claim 8 , wherein the detection in step a) is initiated by the primary device through an established communication channel, for example, a wired or Wi-Fi connection.

10. The method of claim 8 , wherein the detecting in step a) is triggered by detecting an initial acoustic signal transmitted from the secondary device.

11. 9. The method of claim 8, comprising measuring and analyzing background acoustic noise and transmitting the acoustic signal within a time window and / or frequency range having low noise.

12. The method of claim 8 , wherein steps b) to e) are repeated in a predetermined sequence and / or upon occurrence of measured movement of the secondary device.

13. The method of claim 8 , wherein the detection of the presence of a secondary device is initiated by registered activity by the secondary device.

Images