Method and system for supporting multi-display interaction using wearable devices

Abstract

Methods and systems for supporting multi-display interaction using a wearable device are disclosed.

Description

Technical Field

[0001] The present invention relates to a method and system for supporting multi-display user interaction, wherein a wearable device with an inertial sensor (eg, an inertial measurement unit (IMU) or an inertial-magnetic measurement unit (IMMU)) is used. Background Art

[0002] A user can have multiple display devices (e.g., multiple screens) showing content, and can switch attention between different display devices while working. Traditionally, a user must manually (e.g., using an input device such as a mouse) move a cursor between display devices to interact with content on different display devices.

[0003] To help improve the efficiency of user interaction, attempts have been made to track or detect the display device that the user is currently viewing, so that the user does not need to manually move the cursor to the display device of interest. Prior art can use infrared light-emitting diode (LED) markers and / or cameras to track the user's visual attention (e.g., using eye tracking technology). However, such techniques may be expensive to implement (e.g., requiring the use of many expensive infrared markers and cameras) and / or may be computationally complex (which may result in inefficient use of computing resources).

[0004] Therefore, it would be useful to provide improved methods and systems for supporting multi-display interaction. Summary of the invention

[0005] In various examples, the present invention describes methods and systems for supporting multi-display user interactions. A wearable device (e.g., smart glasses, smart headphones, head-mounted displays, etc.) with an inertial sensor (e.g., an IMU or IMMU) is used to detect the user's head posture. The detection of a first set of inertial measurement values ​​can be performed simultaneously with the detection of user interactions associated with a first display device (e.g., user interactions with content displayed on the first display device) to register a mapping between the first set of inertial measurement values ​​and the first display device. Then, when the first set of inertial measurement values ​​is detected again, the mapping can be used to infer user attention and facilitate user interaction on the first display device.

[0006] Examples of the present invention may provide a user with a more intuitive and / or more efficient way to interact with multiple display devices. By registering a mapping between a detected head gesture and a specific display device, the user's visual attention to that specific display device may be determined, which may provide a technical advantage because expensive hardware or complex calculations associated with gaze tracking may be avoided.

[0007] In some examples, different types of user interactions can be supported, including various head-based user interactions. This can provide users with unique ways to interact with content in a multi-display setup.

[0008] In some exemplary aspects, the present invention describes a method comprising: obtaining a first set of inertial measurement values ​​representing movement of a user's head; and when a first user interaction associated with a first display device among multiple display devices is detected, determining a first mapping between the first set of inertial measurement values ​​and the first display device.

[0009] In one example of the above exemplary aspects of the method, the method may further include: storing the first mapping.

[0010] In an example of any of the above-mentioned exemplary aspects of the method, the method may also include: obtaining a subsequent set of inertial measurement values ​​representing subsequent movements of the user's head; identifying a first mapping that matches the subsequent set of inertial measurement values; and based on the identified first mapping, enabling further user interaction with content displayed on the first display device.

[0011] In one example of the above exemplary aspect of the method, implementing further user interaction may include: activating a cursor on the first display device, the cursor being activated at a position corresponding to a previous user interaction on the first display device.

[0012] In an example of the above exemplary aspect of the method, implementing further user interaction may include: detecting a scroll input; and scrolling content displayed on the first display device based on the scroll input.

[0013] In the example of the above exemplary aspect of the method, implementing further user interaction may include: controlling display parameters of the first display device.

[0014] In an example of any of the above-mentioned exemplary aspects of the method, the first user interaction can be one of the following: voice input identifying the first display device; keyboard input for interacting with content displayed on the first display device; mouse input for interacting with content displayed on the first display device; or touch input sensed by a touch sensor of the first display device.

[0015] In an example of any of the above-mentioned exemplary aspects of the method, the method may include: obtaining a second set of inertial measurement values ​​representing movement of a user's head; when a second user interaction associated with a second display device among multiple display devices is detected, determining a second mapping between the second set of inertial measurement values ​​and the second display device; wherein, in response to obtaining another set of inertial measurement values ​​matching the first mapping, user interaction with content displayed on the first display device can be achieved; wherein, in response to obtaining another set of inertial measurement values ​​matching the second mapping, user interaction with content displayed on the second display device can be achieved.

[0016] In an example of the above example aspect of the method, the method may include storing the second mapping.

[0017] In an example of the above-described example aspect of the method, the method may include: detecting a selection of an object displayed on a first display device; obtaining a third set of inertial measurement values ​​representing movement of a user's head; identifying a second mapping that matches the third set of inertial measurement values; and moving the selected object based on the identified second mapping for display on a second display device.

[0018] In an example of any of the above exemplary aspects of the method, the method may include: obtaining another set of inertial measurement values ​​indicating that the user movement is greater than a defined threshold; deleting all stored mappings.

[0019] In an example of any of the foregoing exemplary aspects of the method, multiple display devices may be controlled by a single electronic device.

[0020] In any of the above exemplary aspects of the method, multiple display devices may be controlled by multiple electronic devices.

[0021] In an example of any of the above exemplary aspects of the method, the inertial measurements may be obtained from an inertial sensor of a wearable device worn on or near the user's head.

[0022] In some exemplary aspects, the present invention describes a computing system comprising a processing unit for executing computer-readable instructions to cause the computing system to perform any of the above exemplary aspects of the method.

[0023] In an example of the above exemplary aspect of the computing system, the computing system may include: an inertial sensor for acquiring a set of inertial measurement values; wherein the computing system may be configured to be worn on or near a user's head.

[0024] In some exemplary aspects, the present invention describes a non-transitory computer-readable medium having encoded therein instructions, wherein the instructions are executable by a processing unit of a computing system to cause the computing system to perform any of the above exemplary aspects of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Reference is now made by way of example to the accompanying drawings which illustrate exemplary embodiments of the present application, in which:

[0026] Figure 1 is a schematic diagram of an example of a user interacting with multiple display devices according to an example of the present invention;

[0027] Figure 2 is a block diagram of an example of a setup with multiple display devices according to an example of the present invention;

[0028] Figure 3 is a block diagram of some components of an exemplary computing system according to examples of the present invention;

[0029] Figure 4 is a flow chart of an exemplary method for implementing user interaction in a multi-display setting according to an example of the present invention;

[0030] FIG. 5A to FIG. 5F Shows Figure 4 An exemplary implementation of the method;

[0031] Figure 6 is a block diagram of another example of a setup with multiple display devices according to an example of the present invention.

[0032] The same reference numerals may be used in different drawings to denote the same components. DETAILED DESCRIPTION

[0033] Prior art for tracking a user's visual attention in a multi-display environment typically requires the use of gaze tracking technology. This may require the use of infrared light-emitting diode (LED) markers to mark different display devices and / or the use of infrared cameras to detect and track the user's eyes. In addition to requiring expensive hardware, complex calculations may also be required. For example, in order to achieve tracking of the user's visual attention, each display device may need to be registered in a memory, and then the camera may capture a scene including multiple display devices to register the position of each display device. Then, this information may need to be associated with the data provided by the eye tracking camera. Finally, based on the registration of the display device and the results of the gaze detection algorithm, an approximate gaze point representing the user's visual attention is calculated. As those skilled in the art will appreciate, such prior art may be costly and / or computationally complex, which may limit practical implementation.

[0034] Wearable devices with inertial sensors, such as an inertial measurement unit (IMU) or an inertial-magnetic measurement unit (IMMU), can be used to detect the user's head posture. However, there are difficulties in tracking the user's visual attention using inertial sensors. For example, due to drift and / or error accumulation in the sensor measurements, it is often difficult to obtain accurate measurements of the user's head posture from the inertial sensors over a long period of time. This means that the inertial sensors need to be frequently recalibrated, for example using an external camera, which may be impractical in practical applications.

[0035] The present invention provides an example in which an inertial sensor on a wearable device can be used to detect a user's visual attention to a specific display device in a multi-display setting without the need for additional hardware. In the examples disclosed herein, by inferring the display device that is currently the target of the user's attention, explicit recalibration of the inertial sensor may not be required.

[0036] Figure 1 An example of a user 10 interacting with multiple display devices (particularly a first display device 220a and a second display device 220b (generally referred to as display devices 220)) in a multi-display setting is shown. Although two display devices 220 are shown, it should be understood that this is merely exemplary and that more than two display devices 220 may be present in a multi-display setting. The display devices 220a, 220b may communicate with the same processing unit (e.g., a single desktop computer) or different processing units (e.g., two different laptop computers). The two display devices 220a, 220b may display different visual content, or may display multiple views of the same content (e.g., multiple views of the same software application). The user's visual attention (indicated by the dotted line) may currently be directed toward one display device 220a. In this example, the user 10 wears the wearable device 100 (e.g., headphones) on his head.

[0037] Figure 2 is a block diagram of some exemplary computing hardware that may be used in Figure 1 Used in the examples.

[0038] In this example, each display device 220a, 220b communicates (e.g., via a wired connection) with an electronic device 200 (e.g., a desktop computer). The electronic device 200 also communicates with an input device 210 (e.g., a mouse, keyboard, touch interface, microphone, etc.). The user ( Figure 2 2 (not shown) can use the input device 210 to interact with the content displayed on each display device 220a, 220b.

[0039] For example, the electronic device 200 also communicates with the wearable device 100 using a wireless connection such as Bluetooth or using a wired connection. The wearable device 100 can be any smart device (e.g., an electronic device with wired or wireless communication capabilities) that can be worn on or near the user's head, such as smart glasses, smart headphones, head-mounted displays (head-mounted display, HMD), etc. The wearable device includes an inertial sensor 110 (e.g., IMU or IMMU). The inertial sensor 110 can be any suitable sensor capable of measuring the user's head posture, such as a 9-axis IMMU (e.g., with a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer), or other types of inertial sensors that can have higher or lower accuracy. In the example shown, the wearable device 100 includes a memory 120, which can store instructions for performing at least some of the functions disclosed herein. In other examples, the wearable device 100 can communicate with an external memory (e.g., a memory of the electronic device 200, or a memory of another external device (not shown)).

[0040] Figure 3 is a block diagram of some components of an exemplary computing system 300 (which may also be generally referred to as an apparatus) that may be used to implement embodiments of the present invention. The computing system 300 may be used to perform the methods disclosed herein. The computing system 300 may represent the wearable device 100, the electronic device 200, or another device that communicates with the wearable device 100 and the electronic device 200. Although an exemplary embodiment of the computing system 300 is shown and discussed below, other embodiments may also be used to implement the examples disclosed herein and may include components different from those shown. Although Figure 3 A single instance of each component is shown, but there may be multiple instances of each component shown.

[0041] The computing system 300 includes at least one processing unit 302, such as a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, or a combination thereof. The processing unit 302 may have one or more processor cores.

[0042] The computing system 300 may include an input / output (I / O) interface 304. The I / O interface 304 may be connected to an input device and / or an output device, depending on the embodiment. For example, if the computing system 300 represents the electronic device 200, the I / O interface 304 may be connected to the input device 210 and the display device 220. If the computing system 300 represents the wearable device 100, the I / O interface 304 may be connected to the inertial sensor 110.

[0043] The computing system 300 may include a network interface 306 for wired or wireless communication with a network (e.g., an intranet, the Internet, a P2P network, a WAN and / or a LAN) or other devices. For example, wired or wireless communication between the electronic device 200 and the wearable device 100 may be implemented through the network interface 306.

[0044] The computing system 300 includes a memory 308, which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The non-transitory memory 308 may store instructions executed by the processing unit 302, for example, to perform the examples described in the present invention. For example, the memory 308 may include instructions executable by the processing unit 302 to implement the display registration module 310 discussed further below. The memory 308 may include other software instructions, such as for implementing an operating system and other applications / functionality. For example, the memory 308 may include software instructions for mapping a user's head gesture to an input command, as disclosed herein.

[0045] In some examples, computing system 300 may also include other electronic storage units (not shown), such as a solid state drive, a hard disk drive, a magnetic disk drive, and / or an optical disk drive. In some examples, one or more data sets and / or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with computing system 300), or by a transient or non-transient computer-readable medium. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory. For example, the components of computing system 300 may communicate with each other via a bus.

[0046] In an example of the present invention, inertial measurements from the wearable device 100 (e.g., obtained from the inertial sensor 110 of the wearable device 100) can be used to determine the head posture of the user 10 (e.g., relative to a global reference frame, such as relative to the direction of gravity). In some examples, when the wearable device 100 is active and communicating with the electronic device 200, inertial measurements can be continuously obtained from the wearable device 100. When a user interaction associated with the first display device 220a is detected, the display registration module 310 is used to obtain the current inertial measurement value (which is used as a proxy for the user's head posture) and generate a mapping between the current inertial measurement value and the first display device 220a. The mapping can be stored as an inertial measurement value set associated with the first display device 220a. Then, the first display device 220a can be considered to be "registered" by the display registration module 310. A similar process can be used to generate and store mappings for each other display device 220. The stored mapping can then be used to determine whether the user 10 is viewing a particular display device 220. For example, if the inertial measurement values ​​obtained from the wearable device 100 correspond to the stored inertial measurement values ​​of a given mapping (e.g., the inertial measurement values ​​are within a defined range of the stored inertial measurement values ​​of the given mapping), the content displayed on the display device 220 indicated by the given mapping can be automatically selected for user interaction (e.g., activating a cursor to interact with the content displayed on the display device 220).

[0047] In this way, the inertial sensor 110 can be implicitly calibrated to register the spatial relationship between each display device 220 and the inertial measurement values. User interactions associated with a given display device 220 are used to infer that the user 10 is viewing the given display device 220, and the inertial sensor 110 can be calibrated accordingly with respect to the given display device 220. The user interaction can be any user interaction indicating a particular display device 220, including interaction with content displayed on a particular display device 220 (e.g., through mouse input (e.g., mouse click), through keyboard input (e.g., text input), or through touch input (e.g., touch gestures on a touch-sensitive display device 220)), or interaction selecting a particular display device 220 (e.g., through voice input (e.g., voice input recognized as identifying a particular display device 220), or through touch input (e.g., touch gestures on a touch-sensitive display device 220)), as well as other possibilities. In this way, explicit calibration of the inertial sensor 110 (e.g., requiring a large amount of explicit user input and / or requiring the use of an external camera) may not be required. Further details will be referred to in detail. Figure 4 Give a description.

[0048] Figure 44 is a flow chart of an exemplary method 400 for performing multi-display interaction using inertial measurements obtained from a wearable device. The method 400 may be performed by a computing system (e.g., computing system 300, which may be embodied as electronic device 200, wearable device 100, or another computing device in communication with electronic device 300 and wearable device 100). For example, the method 400 may include the functionality of the display registration module 310 and may be performed by a processing unit that executes instructions to implement the display registration module 310.

[0049] At 402 , method 400 can optionally be initiated by detecting that the wearable device 100 is communicating with the electronic device 200 (eg, a wireless connection has been established and activated between the wearable device 100 and the electronic device 200 ).

[0050] At 404, an inertial measurement set representing the movement of the user's head is obtained. For example, if the inertial sensor 110 of the wearable device 100 is a 9-axis IMMU, the inertial measurement set may include 9 measurements (corresponding to 3 axes of the accelerometer, 3 axes of the gyroscope, and 3 axes of the magnetometer). The inertial measurement set may include a greater or lesser number of measurements, depending on the implementation of the inertial sensor 110.

[0051] How the inertial measurement value set is obtained may depend on the embodiment of the computing system executing the method 400. For example, if the method 400 is being implemented on the wearable device 100, the inertial measurement value set may be obtained by collecting data directly from the inertial sensor 110. If the method 400 is being implemented on the electronic device 200 or another computing device, the inertial measurement value set may be obtained by receiving the inertial measurement value set from the wearable device 100.

[0052] Optionally, at 406, if it is determined that the inertial measurement value set indicates that the user has moved greater than a defined threshold, any registered display devices 220 may be unregistered from the display registration module 310, and any stored mappings between the display devices 220 and the inertial measurement value set may be deleted. For example, if the inertial measurement value set indicates that the user is walking or moving at a speed greater than a defined threshold, this may mean that the user has left the display device 220. Even if the user subsequently returns to the display device 220, a new mapping may need to be generated because the inertial measurement values ​​may have drifted during the time the user was away. After step 406, the method 400 may return to step 404 to continue monitoring the inertial measurement values.

[0053] Optionally, at 408, if there is already a stored mapping that matches the set of inertial measurement values, the mapping can be identified and user interaction can be enabled for display device 220 mapped by the identified mapping. Identifying the mapping may involve querying a table storing the mapping, which may be stored locally or externally.

[0054] A match between a set of inertial measurement values ​​and a stored mapping can be determined if the set of inertial measurement values ​​is within a defined range of inertial measurement values ​​stored for the mapping. For example, if a mapping is stored with a set of inertial measurement values ​​represented by inertial vectors [ax, ay, az, gx, gy, gz, mx, my, mz] (corresponding to 3 axes of an accelerometer, 3 axes of a gyroscope, and 3 axes of a magnetometer), the set of inertial measurement values ​​can be considered a match if the set of inertial measurement values ​​is within a range of + / –10% of each entry in the stored inertial vectors, or other defined ranges. In some examples, the range can be defined based on known or expected dimensions of the display device 220, among other possibilities.

[0055] After identifying the mapping, the display device 220 mapped by the identified mapping can be identified (e.g., each display device 220 can have a unique identifier that can be used to identify each display device 220 in the stored mapping). Then, the content displayed on the identified display device 220 can be automatically selected to enable user interaction. For example, a cursor can be displayed or otherwise activated on the identified display device 220. The cursor can be displayed or activated at a previously saved position in the displayed content (e.g., if the user previously interacted with the content displayed on the identified display device 220, the cursor can be activated again at the position of the last user interaction). Therefore, the user can immediately begin interacting with the content displayed on the identified display device 220 without having to manually select the content.

[0056] In some examples, step 408 can be performed after determining that the inertial measurement value has been relatively stable (e.g., changes less than 5%) within a defined time period (e.g., 0.5s) to avoid inadvertently enabling user interaction while the user is still moving his head. After step 408, method 400 can return to step 404 to continue monitoring the inertial measurement value.

[0057] Returning to step 404 , if the set of inertial measurement values ​​does not indicate that the user has moved beyond the defined threshold, and there is no stored mapping that matches the set of inertial measurement values, step 410 of method 400 is performed.

[0058] At 410, a user interaction associated with a given display device 220 is detected (if no user interaction is detected simultaneously with the acquired set of inertial measurement values, the method 400 may return to step 404). For example, the user interaction may be a user input for interacting with content displayed on a given display device 220 (e.g., via a mouse input, a keyboard input, a touch input, etc.), or a user input for selecting a given display device 220 (e.g., via a voice input for identifying a given display device 220, via an input of a function key for selecting a display device 220, etc.). The detected user interaction indicates that the user's visual attention is currently on the given display device 220. The detected user interaction occurs simultaneously with the acquired set of inertial measurement values ​​in step 404.

[0059] How the user interaction is detected may depend on the embodiment of the computing system executing the method 400. For example, if the method 400 is being implemented on the electronic device 200, the electronic device 200 may directly detect the user interaction and identify the given display device 220 associated with the user interaction. If the method 400 is being implemented on the wearable device 100, the electronic device 200 may directly detect the user interaction and the given display device 220 associated with the user interaction. The electronic device 200 may then send a signal to the wearable device 100 indicating that the user interaction associated with the given display device 220 was detected; in this manner, the wearable device 100 may indirectly detect the user interaction associated with the given display device 220. A similar process may occur if the method 400 is being implemented on another computing device (e.g., a smartphone) that is not the electronic device 200.

[0060] At 412, a mapping is generated to map between the set of inertial measurement values ​​and the given display device 220. The mapping is stored in a memory. The given display device 220 is now considered to be registered by the display registration module 310. In this way, the inertial sensor 110 of the wearable device 100 can be implicitly calibrated relative to the given display device 220. In some examples, the mapping is generated when the inertial measurement values ​​are relatively stable (e.g., change less than 5%) within a defined time period (e.g., 0.5s).

[0061] The generation of the mapping and the storage of the mapping can be performed in different computing systems. For example, if the method 400 is being implemented on the wearable device 100 and the wearable device 100 lacks sufficient storage, the wearable device 100 can transmit the generated mapping to another computing system (e.g., to the electronic device 200 or to another computing device, such as a smartphone) for storage. In another example, if the wearable device 100 has sufficient storage, or if the method 400 is being implemented on the electronic device 200 (or another computing system with memory), the generation and storage of the mapping can be performed on the same computing system.

[0062] In some examples, the mapping may be stored in the form of a table that associates an inertial vector representing a set of inertial measurement values ​​with an identifier of a given display device 220. In some examples, the set of inertial measurement values ​​stored with the mapping may be an average of inertial measurement values ​​acquired over a defined time period (e.g., 0.5) when the inertial measurement values ​​have been relatively stable.

[0063] The user may perform further interactions with the given display device 220. The method 400 may return to step 404 to continue monitoring inertial measurements.

[0064] If the user no longer views the given display device 220 and / or interacts with a different display device 220, and subsequently returns to viewing the given display device 220, the stored mapping may be used to automatically enable user interaction with the given display device 220 again (eg, as described in step 408).

[0065] As described above, the method 400 may be performed by the wearable device 100, the electronic device 200, or another computing device.

[0066] In some examples, method 400 can be performed by wearable device 100. Wearable device 100 can have a processing unit with sufficient processing power to generate a mapping between inertial measurement values ​​and a given display device 220. Wearable device 100 can also have a memory in which the generated mapping can be stored. In other examples, wearable device 100 is capable of generating a mapping between inertial measurement values ​​and a given display device 220, but can rely on an external device (e.g., an electronic device 200 or another computing device in communication with wearable device 100, such as a smartphone) to store the mapping. In other examples, method 400 can be performed by electronic device 200 or another computing device using inertial measurement values ​​obtained from wearable device 100.

[0067] In some examples, each stored mapping may be stored with a timestamp indicating the time when the mapping was generated and stored. Because the inertial sensor 110 may exhibit drift over time, the stored timestamp may be a useful indicator that the stored mapping needs to be updated. For example, the method 400 may include an additional step of checking the timestamp stored with each stored mapping. If the timestamp stored with a particular mapping is earlier than a predefined time period (e.g., 1 minute earlier than the current timestamp), the mapping may be deleted. This may enable the generation and storage of new mappings, thereby enabling the implicit calibration of the inertial sensor 110 to be updated. In step 408, if a stored mapping matching the set of inertial measurement values ​​is identified, it may be determined whether the timestamp of the identified mapping is earlier than the predefined time period. If the timestamp of the identified mapping is within the predefined time period, the mapping may be sufficiently up-to-date, and step 408 may be performed as described above. If the timestamp of the identified mapping exceeds the predefined time period, the identified mapping may be deleted, and step 410 of the method 400 may be performed.

[0068] FIG. 5A to FIG. 5F 4 shows an exemplary implementation of method 400 in a multi-display desktop setting. In this example, method 400 may be performed by electronic device 200, which communicates with first display device 220a and second display device 220b, and also communicates with wearable device 100 worn on or near the head of user 10. To help understand, FIG. 5A to FIG. 5F Schematic diagram 500 is included to help illustrate the mapping between inertial measurements and display device 220 , however, schematic diagram 500 need not be explicitly generated or displayed.

[0069] FIG. 5A to FIG. 5F A common setup is shown in which a user 10 interacts with a desktop computer (e.g., electronic device 200, not shown) through two or more display devices 220 while wearing a wearable device 100 (e.g., headphones) with inertial sensors. The user 10 can interact with content displayed on the display devices 220 using input devices 210 such as a keyboard and a mouse.

[0070] exist Figure 5A In the diagram 500, there is no mapping between any set of inertial measurement values ​​and any display device 220 (represented by the absence of a connection line between the first display and the second display and the inertial measurement values). The table 510 (which may be stored in a memory of the electronic device 200) storing any generated mappings between inertial measurement values ​​(represented by inertial vectors) and display devices may be empty. For example, the user 10 may have just started working and has not yet provided any input or interaction.

[0071] exist Figure 5BIn the embodiment of the present invention, there is a user interaction associated with the first display device 220a. In this case, the user 10 has entered text (e.g., using a keyboard) in a text editing application displayed on the first display device 220a, and a valid text insertion cursor is displayed on the first display device 220a. In other examples, the user interaction may be a touch input, a mouse input, a voice input, etc. At the same time, an inertial measurement value set is obtained from the wearable device 100, which corresponds to the head position of the user when viewing the first display device 220a (indicated by a dotted line). Therefore, a first mapping is generated between the current inertial measurement value set and the first display device 220a (indicated by a new line between the first display and the inertial measurement value in the schematic diagram 500). The first mapping between the current inertial measurement value set represented by the inertial vector v1 and the first display device 220a identified by the device identifier ID_1 is stored in the table 510. In this way, the inertial measurement values ​​acquired by the inertial sensor 110 have been implicitly calibrated relative to the first display device 220a.

[0072] exist Figure 5C , there is a user interaction associated with the second display device 220b. In this case, the user 10 has performed a mouse click or moved a chevron cursor (e.g., using a mouse) on the content displayed on the second display device 220b (it can be noted that the text insertion cursor is no longer active on the first display device 220a). In other examples, the user interaction can be a touch input, a mouse input, a voice input, etc. At the same time, another set of inertial measurement values ​​is obtained from the wearable device 100, which corresponds to the head position of the user when viewing the second display device 220b (indicated by a dotted line). Therefore, a second mapping is generated between the new set of inertial measurement values ​​and the second display device 220b (indicated by a new line between the second display and the inertial measurement values ​​in the schematic 500). The second mapping between the set of inertial measurement values ​​represented by the inertial vector v2 and the second display device 220b identified by the device identifier ID_2 is stored in the table 510. In this way, the inertial measurement values ​​acquired by the inertial sensor 110 have been implicitly calibrated relative to the second display device 220b.

[0073] The first display device 220a and the second display device 220b are now both considered registered.

[0074] exist Figure 5D, the user 10 again turns their head position to view (indicated by the dashed line) the first display device 220a, but the user 10 does not perform any interaction to select the first display device 220a (indicated by the absence of the input device 210). The table 510 of stored mappings is used to identify whether there are any stored mappings that match the current set of inertial measurement values ​​obtained from the wearable device 100. In this case, a match with the first mapping is found (indicated by the thicker connecting line in the schematic 500 and by the thicker outline in the table 510). It can be noted that the first mapping can be identified as matching the current inertial measurement value even if the current inertial measurement value does not exactly match the stored inertial vector v1. For example, a match can be identified if the current inertial measurement value is within a defined margin (e.g., 10%) of the stored inertial vector v1. This can be demonstrated by the user's head position (indicated by the dashed line) in Figure 5B and Figure 5D There are slight differences between them.

[0075] As a result of the identified first mapping, the first display device 220a is identified, and user interaction with the content displayed on the first display device 220a is enabled. For example, the text insertion cursor is activated and displayed again on the first display device 220a. The text insertion cursor may be at the same position as the previous user interaction on the first display device 220a (e.g., Figure 5B 20a) is activated at the same location as shown. Thus, the user can immediately begin interacting with the content on the first display device 220a (e.g., entering more text into a text editing application at the location they previously typed) without having to first explicitly select the first display device 220a using mouse input or keyboard input. Thus, examples of the present invention can enable the user 10 to more easily switch between different display devices 220, based only on the display device 220 they are viewing and without explicitly selecting a target display device 220. For example, efficiency can be improved by reducing the number of explicit inputs that need to be processed.

[0076] exist Fig. 5F , user 10 has left display device 220. Inertial measurements obtained from wearable device 100 indicate that the user has moved beyond a defined threshold. Therefore, display device 220 is deregistered and the stored mapping is deleted (indicated by the removal of the connection line from schematic 500 and table 510 being empty). The stored mapping may be deleted because inertial sensor 110 tends to exhibit drift in inertial measurements over time. Therefore, if user 10 leaves display device 220 and subsequently returns, the previously generated mapping may no longer be valid and a new mapping may need to be generated.

[0077] If the user 10 moves back to the display device 220 again (e.g., back to Figure 5A), a new mapping is generated and stored.

[0078] Optionally, feedback may be provided to the user 10 to indicate that the display device 220 has been registered or that the user's attention has been determined to be on a particular display device 220. For example, a graphical representation similar to the schematic diagram 500 may be displayed (e.g., as a vignette) to show the user 10 whether the display device 220 has been registered. In another example, when the stored mapping matches the current inertial measurement value and a particular display device 220 has been identified based on the matching mapping, the particular display device 220a may be briefly highlighted (e.g., with increased brightness). Other such feedback mechanisms may also be used.

[0079] Although the previous examples have been described in the context of multi-display interactions in which there is a single electronic device 200 communicating with multiple display devices 220, examples of the present invention may also support multi-display interactions in which there are multiple electronic devices 200 communicating with multiple display devices 220. For example, the present invention may implement multi-display interactions in which there are multiple single-screen electronic devices 200 (e.g., a laptop computer and a tablet computer), and multi-display interactions in which there are multiple electronic devices 200, including electronic devices 200 that control two or more display devices 220 (e.g., a tablet computer and a desktop computer, where the desktop computer is connected to two display screens). Different electronic devices 200 may use the same or different input modes (e.g., a tablet computer may support touch input, but a desktop computer may not support touch input).

[0080] Figure 6 is a block diagram of an example of the present invention implemented in a scenario where multiple electronic devices 200 communicate with multiple display devices 220.

[0081] Figure 6 An example similar to Figure 2 In the example of FIG. 1 , however, there are three display devices 220a, 220b, 220c controlled by two electronic devices 200a, 200b. In this example, a first electronic device 200a (e.g., a desktop computer) communicates with a first display device 220a and a second display device 220b, and a second electronic device 200b (e.g., a table) communicates with a third display device 220c. Each electronic device 200a, 200b can receive user input via a corresponding input device 210a, 210b. The two input devices 210a, 210b can support the same or different input modes.

[0082] In this example, the second electronic device 200b communicates with the first electronic device 200a, but does not communicate with the wearable device 100. The communication between the wearable device 100 and the second electronic device 200b (e.g., the transfer of inertial measurement values, detected user interactions, etc.) can be via the first electronic device 200a. In other examples, the second electronic device 200b and the wearable device 100 can communicate directly.

[0083] It should be understood that method 400 may be performed in Figure 6 For example, if the method 400 is implemented on the first electronic device 200a, the first electronic device 200a may receive communications from the second electronic device 200b to detect user interactions associated with the third display device 220c.

[0084] At the same time, the first electronic device 200a may receive an inertial measurement value set from the wearable device 100. Then, the first electronic device 200a may generate and store a mapping between the received inertial measurement value set and the third display device 220c.

[0085] For example, a user can touch the touch screen of a tablet computer while viewing the display screen of the tablet computer, and a user can input mouse clicks while viewing each screen connected to a desktop computer. In another example, a user can provide voice input (e.g., verbal input such as "left screen", "right screen", "tablet") to identify each display device 220 while viewing each corresponding display device 220. In this way, a mapping can be generated and stored to register each display device 220.

[0086] Then at a later time, when the inertial measurements from the wearable device 100 match the stored mapping, the first electronic device 200a can identify the third display device 220c as the target of the user's visual attention and can communicate with the second electronic device 200b to enable the user to interact with content displayed on the third display device 220c.

[0087] In this way, enabling user interactions to be easily transferred between different display devices 220 may also implicitly enable users to easily interact between different electronic devices 200. In some examples, if different electronic devices 200 support a common input mode, different electronic devices 200 may transmit user input to each other, so that user input received at a first input device 210a connected to a first electronic device 200a may be transmitted to a second electronic device 200b. By using head movement to switch between different display devices 220 of different electronic devices 200, rather than having to switch to a different input device 210 to interact with each different electronic device 200, a user may be enabled to interact with different electronic devices 200 through a single input device 210 (which uses a common input mode).

[0088] In some examples, after each display device has been registered (ie, a corresponding mapping has been generated and stored), head-based user interaction can be supported when the user wears the wearable device 100 with the inertial sensor 110.

[0089] In one example, each display device 220 may maintain display of a cursor at the location of the last user interaction, even if the cursor is not active (i.e., even if the cursor is not currently being used for user interaction). Then, when the user's attention is directed to a particular display device 220 (as determined by the inertial measurements matched to the mapping corresponding to the particular display device 220), the user may immediately begin interacting with the content from the location of the now-active cursor.

[0090] In another example, scrolling input (e.g., using a scroll button on a mouse) while the user's attention is on a particular display device 220 (determined by inertial measurement values ​​that match a mapping corresponding to the particular display device 220) may be used as a command to scroll content displayed on the particular display device 220.

[0091] In another example, when the user's attention is on the first display device 220a (determined by the inertial measurement values ​​matched to the mapping corresponding to the first display device 220a), the user can select (e.g., click with a mouse) an object (e.g., an icon, a window, etc.) displayed on the first display device 220a. Then, instead of using a traditional drag input (e.g., using the movement of a mouse) to move the selected object, the user can move their head to drag the selected object. For example, this can be used to drag the selected object within the first display device 220a, or can be used to drag the selected object to a different display device (e.g., to the second display device 220b).

[0092] In another example, when the user's attention is on the first display device 220a (determined by inertial measurements that match a mapping corresponding to the first display device 220a), the user can control a display parameter (e.g., brightness) using defined head movements or defined head gestures (e.g., moving the head up or down).

[0093] In another example, when the user's attention is on the first display device 220a (determined by the inertial measurement value matched with the mapping corresponding to the first display device 220a), the user can type in the text input displayed on the second display device 220b. For example, the user can read the text content displayed on the first display device 220a and type the text content into the text input field displayed on the second display device 220b. The user's visual attention can initially be on the second display device 220b on which the text is entered, and then when continuing to provide text input, the user's visual attention can be turned to the first display device 220a. An overlay or text box can be displayed on the first display device 220a to display a portion of the text being typed (e.g., a portion of the text input displayed on the second display device 220b), so that the user can more easily check their text input without having to switch attention between the first display device 220a and the second display device 220b. It may be noted that such user interactions (wherein the user's visual attention is on one display device 220 while user input is provided on a different display device 220) may occur after the mappings for all display devices 220 are generated and stored. Furthermore, in this example, continued user input associated with the second display device 220b may override activation of the cursor on the first display device 220a (which would otherwise occur when the user directed his visual attention to the first display device 220a without continued user input associated with the second display device 220b).

[0094] In various examples, the present disclosure has described methods and systems that enable inertial sensors of wearable devices (e.g., smart glasses with an IMU or IMMU, smart headphones, etc.) to be implicitly calibrated with display devices in a multi-display setup. This can help improve the efficiency of user interaction, as well as improve the efficiency of processing because less user input may be required.

[0095] Examples of the present invention can be used to generate and store mappings between inertial measurement values ​​and display devices based on user interactions associated with each display device. Possible user interactions include mouse input, keyboard input, touch input, voice input, etc.

[0096] Using the examples disclosed herein, users can more easily switch active cursors between different display devices. When a cursor is activated on a display device, the cursor can automatically locate the location of the last user interaction on the display device without explicit user input.

[0097] Examples of the present invention may be implemented in a multi-display setup, where multiple display devices are controlled by a single electronic device, or multiple display devices are controlled by multiple electronic devices, which may use the same or different input modes.

[0098] Although the present invention describes methods and processes by following a certain sequence of steps, one or more steps in the methods and processes may be omitted or changed as appropriate. Where appropriate, one or more steps may be performed in an order other than the described order.

[0099] Although the present invention is at least partially described in terms of methods, it will be understood by those skilled in the art that the present invention also relates to various components for performing at least some aspects and features of the described methods, whether hardware components, software, or any combination of the two. Accordingly, the technical solution of the present invention can be implemented in the form of a software product. Suitable software products can be stored in a pre-recorded storage device or other similar non-volatile or non-transient computer-readable medium, including DVDs, CD-ROMs, USB flash drives, removable hard disks, or other storage media. The software product includes instructions tangibly stored therein, which enable an electronic device to perform examples of the methods disclosed herein.

[0100] Without departing from the subject matter of the claims, the present invention may be implemented in other specific forms. The described exemplary embodiments are considered in all respects to be illustrative only and not restrictive. Features selected from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations may be understood within the scope of the present invention.

[0101] In addition, all values ​​and subranges within the disclosed range are also disclosed. In addition, although the disclosed and illustrated systems, equipment and processes herein may include a specific number of elements / assemblies, these systems, equipment and assemblies may be modified to include more or less such elements / assemblies. For example, although any disclosed element / assembly may be a single quantity, the embodiments disclosed herein may be modified to include a plurality of such elements / assemblies. The subject matter described herein is intended to encompass and include all suitable technical changes.

Claims

1. A method, characterized in that include: obtaining a first set of inertial measurements representing motion of a user's head; When a first user interaction associated with a first display device of the plurality of display devices is detected, a first mapping between the first set of inertial measurement values ​​and the first display device is determined.

2. The method according to claim 1, characterized in that Also includes: The first mapping is stored.

3. The method according to claim 1 or 2, characterized in that: Also includes: obtaining a subsequent set of inertial measurements representing subsequent movement of the head of the user; identifying the first mapping that matches the subsequent set of inertial measurements; Based on the first mapping of the identifier, further user interaction with the content displayed on the first display device is achieved.

4. The method according to claim 3, characterized in that Implementing the further user interaction includes: A cursor on the first display device is activated, the cursor being activated at a position corresponding to a last user interaction on the first display device.

5. The method according to claim 3, characterized in that: Implementing the further user interaction includes: Detect scroll input; Content displayed on the first display device is scrolled based on the scroll input.

6. The method according to claim 3, characterized in that: Implementing the further user interaction includes: Controlling display parameters of the first display device.

7. The method according to any one of claims 1 to 6, characterized in that The first user interaction is one of the following: Identifying a voice input of the first display device; keyboard input for interacting with content displayed on the first display device; Mouse input for interacting with content displayed on the first display device; or The touch sensor of the first display device senses a touch input.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: obtaining a second set of inertial measurements representing movement of the head of the user; when a second user interaction associated with a second display device of the plurality of display devices is detected, determining a second mapping between the second set of inertial measurement values ​​and the second display device; wherein, in response to acquiring another set of inertial measurement values ​​matching the first mapping, enabling user interaction with content displayed on the first display device; Wherein, in response to acquiring another set of inertial measurement values ​​matching the second mapping, user interaction with content displayed on the second display device is implemented.

9. The method according to claim 8, characterized in that Also includes: The second mapping is stored.

10. The method according to claim 8 or 9, characterized in that: Also includes: detecting a selection of an object displayed on the first display device; acquiring a third set of inertial measurements representing movement of the head of the user; identifying the second mapping that matches the third set of inertial measurements; The selected object is moved based on the identified second mapping for display on the second display device.

11. The method according to any one of claims 1 to 10, characterized in that Also includes: obtaining another set of inertial measurement values ​​indicating that the user has moved greater than a defined threshold; Delete all storage mappings.

12. The method according to any one of claims 1 to 11, characterized in that The multiple display devices are controlled by a single electronic device.

13. The method according to any one of claims 1 to 11, characterized in that The plurality of display devices are controlled by a plurality of electronic devices.

14. The method according to any one of claims 1 to 13, characterized in that Inertial measurements are obtained from inertial sensors of a wearable device worn on or near the head of the user.

15. A computing system, characterized in that: include: A processing unit configured to execute computer-readable instructions to cause the computing system to perform the method according to any one of claims 1 to 14.

16. The computing system according to claim 15, characterized in that: Also includes: an inertial sensor, the inertial sensor being used to obtain the inertial measurement value set; Wherein, the computing system is used to be worn on or near the head of the user.

17. A non-transitory computer readable medium, characterized in that Instructions are encoded therein, which are executable by a processing unit of a computing system to cause the computing system to perform the method according to any one of claims 1 to 14.

Images