Computer, software module arrangement, circuit arrangement, user device, and method for an improved user interface for simultaneously controlling multiple applications - Patents.com

Abstract

A user device (100) comprising at least one side sensor (104-3) and a controller (101), the side sensor (104-3) configured to accept touchless user input thereby providing a touchless input area (104-4), and the controller (101) configured to detect a group of objects (F) in the touchless input area (104-4), determine a number of objects in the group of objects, determine an application associated with the number of objects, accept input based on the detected group of objects (F), and determine an action associated with the input for the application associated with the number of objects.

Description

[Technical field]

[0001] This application incorporates the entire contents of commonly assigned patent application entitled "COMPUTER, SOFTWARE MODULE ARRANGEMENT, CIRCUIT ARRANGEMENT, USER DEVICE AND METHOD FOR IMPROVED AUGMENTED USER INTERFACE," filed concurrently herewith.

[0002] The present invention relates to an arrangement, an arrangement including a computer, a software module, an arrangement including a circuit, a user device, and a method for providing an improved and enhanced user interface, and in particular to an arrangement, an arrangement including a computer, a software module, an arrangement including a circuit, a user device, and a method for providing an improved and enhanced user interface that allows control of multiple applications, e.g. background applications. [Background technology]

[0003] Today's user equipment or mobile devices have touch screens. They are based on touching the display with one or more fingers, for example on an iPhone, touching the screen and moving a finger from the bottom to the top of the screen can have a certain meaning, moving the finger in a certain way, so that active applications are listed as window-based icons in the screen and the user can scroll between them as a next step.

[0004] Sensors added to such mobile devices include, for example, radar that allows recognition or detection of movement gestures in specific parts of the space surrounding the device.

[0005] In many applications or navigation scenarios, touch input is limited by the size of the device display, but for example scrolling the content itself is not relevant to the size of the device display. For example, smart watches have a very limited physical size of the display.

[0006] In many usage scenarios, with many mobile device applications, when an application becomes active it takes over the entire screen, which potentially means that an area on the touch screen is no longer available for controlling other applications, and controlling other applications often implies a series of multiple touches to navigate to the other application, i.e., the currently displayed application needs to be changed and the other application must be selected to be displayed on the screen.

[0007] Although there are techniques to identify gestures or movements external to the physical device, for example by radar or cameras, these are typically not intuitive from a usage standpoint.

[0008] Modern devices also suffer in that they offer a relatively limited set of possible inputs, for example limiting the number of gestures that are easy to learn and perform, severely limiting usability when running multiple applications at once. Summary of the Invention

[0009] As mentioned above, the inventors have recognized that because touchless input areas are not limited by the same physical boundaries as touchscreens, there are other instrumental areas available for identifying various inputs, such as various gestures. The solution taught in this disclosure allows a user to associate the same gesture or other input with different applications, thereby allowing the same input to be used simultaneously for different applications by associating some fingers with a particular application. Moreover, the solution provided is inventive in itself, simple and elegant.

[0010] As alluded to above, the present invention is based on associating certain fingers with specific applications, whereby it is possible to identify which application the same input is intended for by simply determining how many fingers are used to provide the input.

[0011] It is therefore an object of the present teachings to overcome or at least alleviate or reduce the problems discussed above.

[0012] According to one aspect, a user device is provided, the user device comprising at least one side sensor and a controller, the side sensor configured to accept touchless user input, thereby providing a touchless input area, and the controller configured to detect a group of objects within the touchless input area, determine a number of objects in the group of objects, determine an application associated with the number of objects, accept input based on the detected group of objects, and determine an action associated with the input for the application associated with the number of objects.

[0013] This allows a user to interact in an easy and natural way with a user device that controls multiple applications with a limited user interface and available command set.

[0014] According to some embodiments, the controller is configured to determine a first number of objects and associate the first number of objects with the first application, and to determine a second number of objects and associate the second number of objects with the second application.

[0015] According to some embodiments, the at least one side sensor is configured to distinguish between different types of objects, and the controller is further configured to determine types of objects within the group of objects and to determine a number of objects and applications associated with the types of objects.

[0016] According to some embodiments, the controller is further configured to determine an order of the two types of objects in the group of objects, and to determine an application associated with the number of objects, the types of objects, and the order of the objects.

[0017] According to some embodiments, the controller is further configured to determine that the detected object is a user's finger.

[0018] According to some embodiments, the controller is further configured to determine that the detected object is a pen or stylus.

[0019] According to some embodiments, the first application is a top application and the other application is a background application, which allows for simultaneous control of the top application in the normal manner (using one finger) while allowing control of the background application by simply performing input using two or more fingers, thus providing multiple control without the need to swap or otherwise switch or activate applications.

[0020] According to some embodiments, the first application is a top application and at least the second application is also a top application, which allows simultaneous control of different top applications using the same gesture set.

[0021] According to some embodiments, the controller is further configured to determine a number of objects in the group of detected objects by determining a location area of ​​the group of detected objects.

[0022] According to some embodiments, the controller is further configured to determine a number of objects in the group of detected objects by detecting at least one individual object.

[0023] According to some embodiments, the controller is further configured to accept an input by detecting a gesture and determine, for an application associated with the number of objects, an associated action by executing a command associated with the detected gesture.

[0024] According to some embodiments, the controller is further configured to detect positions and distances of the group of detected objects, accept input by detecting a movement of the group of detected objects, and determine an associated action by operating for an application associated with the number of objects according to the detected movement.

[0025] According to some embodiments, the controller is further configured to act on an application associated with a number of objects according to the detected movement by performing an action associated with an option displayed at a position corresponding to the object when movement is detected towards the user device, determining a new option corresponding to a new position of the object when movement is detected along the user device, canceling at least one option when movement is detected away from the user device, and executing a command when an option is associated with such command.

[0026] According to some embodiments, the user device comprises a display, and at least one of the at least one side sensors is disposed adjacent to the display, the side sensor configured to accept touchless user input at a side of the display, thereby providing a touchless input area at the side of the display.

[0027] According to some embodiments, a user device includes an input device, and at least one of the at least one side sensors is disposed adjacent to the input device, the side sensor configured to accept touchless user input at a side of the input device, thereby providing a touchless input area at the side of the input device.

[0028] According to some embodiments, the user device comprises a side sensor by being operatively connected thereto.

[0029] According to some embodiments, the user device is a smartphone, a smartwatch, or a tablet computer.

[0030] According to some embodiments, the user device is a computer configured to be operatively connected to a keyboard, a cursor control device, and / or a display.

[0031] According to one aspect, a method for use in a user device having at least one side sensor configured to accept touchless user input, thereby providing a touchless input area, the method comprising: detecting a group of objects in the touchless input area; determining a number of objects in the group of objects; determining an application associated with the number of objects; accepting input based on the detected group of objects; and determining an action associated with the input for the application associated with the number of objects.

[0032] According to one aspect, a computer readable medium is provided that carries computer instructions that, when loaded into and executed by a controller of a user device, enable the user device to perform any of the methods of the present disclosure.

[0033] According to one aspect, a software module arrangement is provided for a user device having at least one side sensor configured to accept touchless user input, thereby providing a touchless input area, the software module arrangement comprising a software module for detecting a group of objects in the touchless input area, a software module for determining a number of objects in the group of objects, a software module for determining an application associated with the number of objects, a software module for accepting input based on the detected group of objects, and a software module for determining an action associated with the input for the application associated with the number of objects.

[0034] According to one aspect, there is provided an arrangement adapted for use in a user device having at least one side sensor configured to accept touchless user input, thereby providing a touchless input area, the arrangement comprising: circuitry for detecting a group of objects within a touchless input area; circuitry for determining a number of objects within the group of objects; circuitry for determining an application associated with the number of objects; circuitry for accepting input based on the detected group of objects; and circuitry for determining an action associated with the input for an application associated with the number of objects.

[0035] The solution may be implemented as a software solution, a hardware solution, or a mixture of software and hardware components. Further embodiments and advantages of the invention are given in the detailed description. [Brief description of the drawings]

[0036] Embodiments of the present invention are described below with reference to the accompanying drawings which show non-limiting examples of how the inventive concepts may be put into practice.

[0037] [Figure 1A] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0038] [Figure 1B] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0039] [Figure 1C] 1 shows a schematic diagram of a subsection of a user device, such as the user device of FIG. 1A or FIG. 1B, according to some embodiments of the present invention.

[0040] [Figure 2A] 1B illustrates the user device of FIG. 1A according to some embodiments of the present invention.

[0041] [Figure 2B] 1A also illustrates a user device of FIG. 1A in accordance with some embodiments of the present invention.

[0042] [Figure 3A] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0043] [Figure 3B] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0044] [Figure 3C] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0045] [Figure 3D] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0046] [Figure 4A] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0047] [Figure 4B] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0048] [Figure 4C] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0049] [Figure 5A] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0050] [Figure 5B] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0051] [Figure 5C] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0052] [Figure 5D] 1 shows a schematic diagram of a user device according to some embodiments of the present invention.

[0053] [Figure 6] 1 shows a flowchart of a general method according to some embodiments of the present invention.

[0054] [Figure 7] 1 shows a component diagram of a software module arrangement in accordance with some embodiments of the teachings of the present disclosure.

[0055] [Figure 8] 1 shows a component diagram of an arrangement comprising circuitry according to some embodiments of the teachings of the present disclosure.

[0056] [Figure 9]1 shows a schematic diagram of a computer-readable medium carrying computer instructions that, when loaded into and executed by a controller of an arrangement, enable the arrangement to implement several embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0057] 1A shows a schematic diagram of a user device 100 according to some embodiments of the present invention. According to some exemplary embodiments, the user device 100 is a smartphone, a smartwatch, or a tablet computer. The user device 100 comprises a controller 101, a memory 102, and a user interface 104 (comprising one or more interface components 104-1 to 104-4, as described in more detail below).

[0058] Note that user equipment 100 may comprise a single device or may be distributed across several devices and apparatus.

[0059] The controller 101 is configured to control the overall operation of the user device 100. According to some embodiments, the controller 101 is a special-purpose controller. According to some embodiments, the controller 101 is a general-purpose controller. According to some embodiments, the controller 101 is a combination of one or more of a special-purpose controller and / or a general-purpose controller. As will be appreciated by those skilled in the art, there are many alternatives for how to implement the controller, and field programmable gate array circuits, ASICs, CPUs, GPUs, NPUs, etc. may additionally or alternatively be used. For purposes of this application, all such possibilities and alternatives will be referred to simply as the controller 101.

[0060] The memory 102 is configured to store data such as application data, settings, and computer readable instructions that, when loaded into the controller 101, indicate how the user device 100 should be controlled. The memory 102 may comprise several memory units or devices, which are recognized as being part of the same overall memory 102. There may be one memory unit for the display arrangement to store instructions and application data, one memory unit for the display arrangement to store graphics data, one memory for the communication interface 103 to store settings, and so on. As one skilled in the art will appreciate, there are many possibilities to choose where data should be stored, and thus a generic memory 102 for the user device 100 is considered to comprise any and all such memory units for the purposes of this application. As one skilled in the art will appreciate, there are many alternatives of how to implement memory, for example, using non-volatile memory circuits such as EEPROM memory circuits, or using volatile memory circuits such as RAM memory circuits. For the purposes of this application, all such alternatives will simply be referred to as memory 102.

[0061] According to some embodiments, the user equipment 100 may further comprise a communication interface 103. The communication interface 103 may be wired and / or wireless. The communication interface 103 may comprise several interfaces.

[0062] According to some embodiments, the communication interface 103 includes a USB (Universal Serial Bus) interface. According to some embodiments, the communication interface 103 includes an HDMI (High-Definition Multimedia Interface) interface. According to some embodiments, the communication interface 103 comprises a DisplayPort interface. According to some embodiments, the communication interface 103 comprises an Ethernet interface. According to some embodiments, the communication interface 103 comprises a MIPI (Mobile Industry Processor Interface) interface. According to some embodiments, the communication interface includes an analog interface, a CAN (Controller Area Network) bus interface, an I2C (Inter Integrated Circuit) interface, or other interface.

[0063] According to some embodiments, the communication interface 103 comprises a radio frequency (RF) communication interface. According to one such embodiment, the communication interface 103 may include a Bluetooth interface, a WiFi interface, a ZigBee interface, a Z-Wave™ interface, an RFID™ (radio frequency identifier) ​​interface, a wireless display (WiDi) interface, a Miracast interface, and / or other RF interfaces commonly used for short-range RF communication, etc. According to alternative or supplemental such embodiments, the communication interface 103 comprises a cellular communication interface, such as a fifth generation (5G) cellular communication interface, a Long Term Evolution (LTE) interface, a Global System Mobile (GSM) interface, and / or other interfaces commonly used for cellular communication. According to some embodiments, the communication interface 103 is configured to communicate using the Universal Plug and Play (UPnP) protocol. According to some embodiments, the communication interface 103 is configured to communicate using the Digital Living Network Appliance (DLNA) protocol.

[0064] According to some embodiments, the communication interface 103 is configured to enable communication via two or more of the exemplary technologies given above. The communication interface 103 may be configured to enable the user equipment 100 to communicate with other devices, such as other smartphones.

[0065] The user interface 104 has one or more interface components 104-1 to 104-4, e.g., one or more output devices and one or more input devices. An example of an output device is a display arrangement such as a display 104-1, one or more light sources (not shown in FIG. 1A), and a speaker (not shown). An example of an input device is one or more buttons 104-2, a camera (not shown), and a microphone (not shown). According to some embodiments, the display arrangement comprises a touch display 104-1 that functions as both an output and an input device, capable of both presenting graphical data and accepting input via touch, e.g., via a virtual button.

[0066] The user interface 104 of the user device 100 according to the teachings of the present disclosure further comprises one or more side sensors 104-3 configured to remotely detect and determine the presence of and distance to an object without contact. Such side sensors 104-3 provide an extended user interface area 104-4 where touchless input may be provided. In the example of FIG. 1A, there are two side sensors 104-3A and 104-3B disposed within the user device, one on either side of the display 104-1.

[0067] As mentioned above, user equipment 100 may comprise a single device or may be distributed across several devices and equipment. It should again be noted that although the embodiment described in relation to FIG. 1A shows the user equipment to be a single device, the user equipment may comprise several devices and all embodiments relate to single device user equipment as well as user equipment comprising several devices.

[0068] FIG. 1B shows a schematic diagram of a user device 100 according to some embodiments of the present invention. According to some exemplary embodiments shown in FIG. 1B, the user device 100 is a general user equipment comprising several devices. The user device may be a computer (laptop or desktop) 100-1 connected to or comprising a keyboard 100-2 or other input means (which may be, for example, a cursor control device (e.g., a computer mouse)). It should be noted that even if only two devices 100-1 and 100-2 are shown in FIG. 1B, any number of devices may be utilized. The various devices may be connected to or included with each other directly or indirectly (e.g., connecting a cursor control device to a keypad connected to a desktop computer). The connections may be internal, external, wireless, and / or wired. In the description of the present disclosure, it makes no difference how the devices 100-1 and 100-2 of the user device 100 are connected to each other, it is only important that they are operably connected to each other. Thus, the configuration should be considered to include operably connected, since the devices form part of the user device, especially when in use.

[0069] An example of a user equipment 100 having two or more devices is a computer (a desktop computer is the first device 100-1) with a keyboard (the second device 100-2) arranged with a touchless user interface 104-4 for manipulating objects and commands in the vicinity of the keyboard. Thus, the keyboard is an example of an input device as the second device 100-2. Another example is a cursor control device such as a mouse or a joystick or a touchpad. Alternatively, a display 104-1 for a computer may be arranged with a touchless user interface 104-4 for manipulating objects and commands in the vicinity of the display. If the display is a touch display, the display is an example of both a display and n input devices.

[0070] As an example of FIG. 1A, the user device 100 comprises a controller 101, a memory 102, a user interface 104 and several other components, which will not be described again but simply refer to the above disclosure.

[0071] Thus, according to some embodiments, a user device may be a computer configured to be operatively connected to a keyboard, a cursor control device, and / or a display.

[0072] Figure 1C shows a schematic diagram of components of a user device 100 such as the user device 100 of Figure 1A or Figure 1B, but in the example of Figure 1B there are four side sensors 104-3A-D, one on each side of the display 104-1, thereby providing an enhanced user interface 104-4 having four subsections 104-4A-D. Even though the example shown in Figure 1B focuses on four side sensors and four subsections, there may be variations having a different number of side sensors 103, as well as a different number of subsections.

[0073] It should be noted that the side sensor 104-3 may comprise one sensor or an array of sensors, depending on the technology used to implement such a side sensor. Examples of technologies for implementing such a side sensor are radar sensors, light-based proximity sensors, capacitive-based proximity sensors, to name a few.

[0074] Utilizing such a side sensor 104-4, a user device 100 according to the teachings of the present disclosure is able to determine that an object, such as a user's finger F, is at a distance D from a position L relative to the display 104-1, as shown in FIG. 1B.

[0075] For purposes of the teachings of this disclosure, all sensors are treated as one sensor 104-3 that provides an expanded user interface area for touchless input, hereafter referred to as touchless input area 104-4.

[0076] According to some embodiments in which the display 104-1 is a touch screen, the combination of the touch screen 104-1 and the side sensor 104-3 thus provides a combined interface area having one portion for touch input (the touch display 104-1) and one portion for touchless input (the touchless input area 104-4).

[0077] According to some embodiments, the touchless input area 104-4 is assigned to an application or process running as a top or active application. According to some embodiments, the touchless input area 104-4 is assigned to an application or process running as a background application. According to some embodiments, the touchless input area 104-4 is assigned to at least one application or process running as a background application and at least one application or process running as an active application. As is known, the user device 100 can run applications in an active (or top) mode or a background mode. An application running in an active mode is assigned at least a portion of the display for providing graphical output, including any virtual buttons displayed on the display 104-1 if the display 104-1 is a touch screen, and most of the buttons 104-2 of the user device 100. However, an application running in a background mode is assigned only a portion of the display space, if any, at most, and usually does not control any space at all. To control such a background application, the background application must be activated and made the active or top application, and any other application that was previously running as the active (top) application must be suspended, as described above in the background section. Alternatively, the background application may require multiple inputs and be controlled by pulling up a specific control window, obscuring the currently running active application. However, the user device 100 according to the present teachings is configured to accept inputs related to the application also via the touchless input area 104-4. As described above, the side sensor 104-3 can determine the distance D to the object F.Thus, by determining that the distance D to the object falls below a threshold distance, it is possible to determine activation of a command such as a (touchless) press. Furthermore, as described above, the side sensor 104-3 can determine the position L of the object F relative to the side sensor. Thus, it is possible to assign different commands or controls to different positions along the display 104-1. In this way, the user interface 104 is expanded, allowing for more user control without obscuring or interfering with the display space allocated to the active application.

[0078] Because the side sensor 104-3 can determine position and distance, the side sensor 104-3 can also determine or detect gestures being made by the detected object, and thus the touchless user interface 104-4 may also be used to accept gesture-based input, where the gesture is associated with an action, such as a command, to be performed.

[0079] FIG. 2A shows the user device 100 of FIGS. 1A and 1B, where the controller 101, in combination with the side sensor 104-3, is configured to detect a number of objects F1, F2, Fn and determine the number of objects.

[0080] According to some embodiments, the controller is configured to determine a location area corresponding to where the object is detected, according to this embodiment, multiple objects may be detected even when presented as a single object, such as when held together.

[0081] The size of the location area La is determined and compared to the known size of the objects. Based on this comparison, the number of objects can be readily determined.

[0082] According to some embodiments, the known size of the object is a default size. According to some embodiments, the known size of the object is a calibrated size, such as by user input or measurement of the object.

[0083] Figure 2B also shows the user device of Figures 1A and 1B, where the controller 101, in combination with the side sensor 104-3, is configured to detect multiple objects F1, F2, Fn and determine the number of objects in an alternative or additional manner.

[0084] According to some embodiments, the controller 101 is configured to detect a number of objects F1, F2, Fn and to determine the position of each of the detected objects.

[0085] Combinations of the embodiments disclosed in connection with Figures 2A and 2B allow for combinations in which some objects are held together and other objects are separated.

[0086] It should be noted that although the description herein shows three objects, this is merely an example and any number of objects could be employed: 2, 3, 4, 5 or more.

[0087] According to some embodiments, the object is a user's finger. According to some embodiments, the object is a stylus or pen. According to some embodiments, the object is a stylus or pen, and another object is a user's finger. This allows a user to use a stylus for input for one application and control different applications by placing one or more fingers along the stylus.

[0088] According to some embodiments, the side sensor(s) (either by itself or in combination with the controller) can distinguish between a stylus (or other object) and a finger. This can be utilized to distinguish based not only on the number of fingers (or objects) being used, but also on the type of object. For example, if a stylus and a finger are used, one application may be associated, and if two fingers are used, another application may be associated.

[0089] According to some embodiments, different types of objects / finger orders may be utilized to associate different associations, for example, holding the stylus between the thumb and index finger may be associated with one application, while holding the stylus between the index finger and middle finger may be associated with another application.

[0090] FIG. 3A illustrates the user device 100 of FIGS. 1A, 1B, and 2A, with different commands assigned to different portions of the touchless input area 104-4. In this example, there are three commands C1-C3 assigned to each of the portions P1-P3 of the touchless input area 104-4. It is possible for the user device 100 to determine which command C1-C3 to execute by determining in which portion P1-P3 an object is detected to be below a threshold distance. According to some embodiments, the different commands C1-C3 relate to the same application. According to some embodiments, there is at least one command associated with a first application and at least one command associated with a second application. According to one such embodiment where there are first and second applications, the first application is associated with a first portion of the touchless input area 104-4 and the second application is associated with a second portion of the touchless input area 104-4, and the commands associated with each application are assigned to a sub-portion of the portion of the corresponding application. For example, in the example of FIG. 1C, a first application is assigned portions P1 and P2 and commands C1 and C2 are associated with the first application, and a second application is assigned portion P3 and command C3 is associated with the second application.

[0091] It should be noted that the size of the moieties, as well as the location and / or distribution of the moieties, do not have to be equal or regular, but any distribution is possible.

[0092] To provide an intuitive and easy to learn user interface that is easy to use, the user device 100 according to the teachings of the present disclosure is configured to allow a user to set up the touchless input area 104-4.

[0093] Also shown in Figure 3A is how the position L is determined for a group of detected objects. According to some embodiments, the position L is considered to be the middle of the group of objects. According to one such embodiment, the middle is determined to be the middle of the area position L. According to an alternative embodiment, the middle is considered to be the center of the middle object.

[0094] Since it is possible to determine the distance D to the objects, the variance of the distances can also indicate the number of objects. According to some such embodiments, the number of minimum values ​​of the distance D corresponds to the number of objects.

[0095] Thus, by determining the position of the central object, it is also possible to determine the position of the central object even when the objects are held together.

[0096] If there is an even number of objects and two of them are in the middle, the center between the two middle objects is taken as the position of the middle object, alternatively the middle object is the top of the two middle objects.

[0097] FIG. 3B shows an alternative example where the position of a group of objects is understood as the top object.

[0098] In this context, the top object is seen as the topmost detected object with respect to screen 104-1.

[0099] Figures 3C and 3D show corresponding situations of how to determine a position L for a group of individually detected objects, where Figure 3C shows the selection of the central object as the object that will determine the position L and Figure 3D shows the selection of the top object as the object that will determine the position L.

[0100] The subsection can be determined by finding which subsection the position L corresponds to.

[0101] Selection of the subsection can be detected by detecting and determining that the distance D to the detected object falls within a threshold distance.

[0102] It is therefore possible to determine which subsection S1, S2, Sm (with m indicating the number of subsections) the user has pointed to and selected, thereby enabling the user device to perform an associated action, such as executing a command C1, C2, Cm associated with the subsection.

[0103] As shown, the touchless input area 104-4 is configured to have at least one subsection. In the context of the teachings of the present disclosure, unless otherwise specified, there is no distinction between the touchless input area 104-4 and the subsection. A subsection may also have further subsections associated therewith.

[0104] 3A-3D, three subsections are shown, but it should be understood that any number of subsections may be provided, which are associated with commands for an application, and the subsection numbering may start at 1 (or even 0). A zero subsection allows only gesture input for that application.

[0105] As mentioned above, the controller 101 is configured not only to detect a group of objects, but also to determine the number of objects in the group. Thus, the controller 101 is configured to determine the number of detected objects.

[0106] Also, as described above, the controller 101 is configured to associate a first application with a first number (e.g., 1) of objects and a second number (e.g., 2) of objects with a second application. Thus, the controller 101 is configured to be able to associate commands of a subsection with applications associated with the number of detected objects, thereby varying which command is executed when a press at a location is detected depending on the number of objects performing a selection movement.

[0107] Figure 4A shows a schematic diagram of a user device 100 as in Figure 1A, where a first number of one object F1 has been detected. Subsections S1, S2, and Sm are associated with commands (or actions) for a first application, and the commands are exemplified as C1.1 (first command for the first application), C1.2 (second command for the first application), and C1.m (mth command for the first application). Since m is the number of available subsections, m is also the number of available commands.

[0108] Figure 4B shows a schematic diagram of the user device 100 as in Figure 4A, where a second number of objects, in this case two, F1, F2, have been detected. Subsections S1, S2, and Sm are then associated with commands (or actions) for a second application, the commands being exemplified as C2.1 (first command for the second application), C2.2 (second command for the second application), and C2.m (mth command for the second application).

[0109] Figure 4C also shows a schematic diagram of user device 100 as in Figures 4A and 4B, where a third number, in this case n objects F1, F2,...,Fn, have been detected. Subsections S1, S2, and Sm are then associated with commands (or actions) for the nth (or further) application, where the commands are exemplified as C2.1 (first command for the nth application), C2.2 (second command for the nth application), and C2.m (mth command for the nth application).

[0110] Note that the number of subsections m may vary from application to application, so a first application may be associated with a different number of subsections than a second application.

[0111] 4A, 4B, and 4C, the controller 101 is configured to associate a first application with a first number (e.g., 1) of objects and associate a second number (e.g., 2) of objects with a second application. Also, as described above, the controller 101 can utilize the detected objects to detect gestures being input or provided by a user.

[0112] Thus, the controller 101 may also or alternatively be configured to associate detected gestures with different applications and commands for those applications based on the number of objects detected, thereby varying which command is executed when gesture G is detected depending on the number of objects performing gesture G.

[0113] FIG. 5A shows a schematic diagram of a user device 100 as in FIG. 1A (and FIG. 4A), where a first number of one object F1 has been detected. The available gestures are associated with commands (or actions) for a first application, which are graphically illustrated by a table of gestures and associated commands, such as G1.1-C1.1 (first command for first gesture for first application), G1.2-C1.2 (second command for second gesture for first application), and G1.p-C1.p (th command for pth gesture for first application). The number p is the number of gestures available for a given application, and p is also the number of commands available for a given application. Thus, the number p may be different or the same for each application.

[0114] Figure 5B shows a schematic diagram of the user device 100 as in Figure 5A, where a second number (e.g., two) of objects F1, F2 have been detected. The available gestures are associated with commands (or actions) for a second application, and the gestures and commands are exemplified as G2.1-C2.1 (first command for first gesture of the second application), G2.2-C2.2 (second command for second gesture of the second application), and G2.p-C2.p (pth command for pth gesture of the second application).

[0115] 5C also shows a schematic diagram of the user device 100 as in FIGS. 5A and 5B, where a third number, in this case n, of objects F1, F2, .., Fn have been detected. The available gestures are then associated with commands (or actions) for a second application, and the gestures and commands are illustrated as Gn.1-Cn.1 (first command of the first gesture of the nth application), Gn.2-Cn.2 (second command of the second gesture of the nth application), and Gn.p-Cn.p (pth command of the pth gesture of the nth application).

[0116] Note that some or all of the gestures for different applications may be the same gesture, where the same gesture causes different commands to be executed for different applications based on the number of fingers performing the gesture. Thus, G1.X may be equal to G2.X (where X is from 1 to n) (a gesture for a first application is the same as a gesture for a second application).

[0117] FIG. 5D shows a schematic diagram of a user device 100 as in combination with any of FIGS. 4A-4C and any of FIGS. 5A-5C, where the controller 101 is configured to both assign subsections to commands from one or more applications and to accept gesture input for the one or more applications.

[0118] Thus, it is possible to execute different commands for different applications based on the number of objects used to provide input. As indicated by the subscript x of the gesture number p (i.e., px), each application can have a different number of commands associated with it, and they need not have all, or even some, of the same number of associated commands.

[0119] Note that no specific additional input is required to select or otherwise indicate the application for which the command is intended, and the number of objects indicates the application to use, not the command to be given.

[0120] It should also be noted that the number of applications, n, may vary from implementation or setup to implementation or setup, and even though n is illustrated as 3 in Figures 4C and 5C, it may be any number as discussed above.

[0121] Also, note that even though the first number is illustrated as 1, it could be any other number, and even though the second number is illustrated as 2, it could be any other number.

[0122] According to some embodiments, the first application is an application (or process) running as a top application or an active application. According to some embodiments, the first application is an application (or process) running as a background application. According to some embodiments, the second application is an application (or process) running as a top application or an active application. According to some embodiments, the second application is an application (or process) running as a background application. According to some embodiments, the nth application is an application (or process) running as a top application or an active application. According to some embodiments, the nth application is an application (or process) running as a background application.

[0123] FIG. 6 illustrates a flowchart of a general method according to the present disclosure performed by the controller 101 of the user device of any of the previous or subsequent embodiments. The controller 101 detects (610) that one or more objects are within the touchless input area 104-4. According to some embodiments, the controller 101 is further configured to determine that at least one of the detected objects is a user's finger. According to some embodiments, the controller 101 is further configured to determine that at least one of the detected objects is a user's stylus or pen. The controller 101 determines (620) the number of detected objects and determines (630) an associated application that will be a target application for any input received by the detected object. According to an alternative embodiment (as will be described below), a selection is presented (635).

[0124] The controller 101 then accepts 640 the input and determines 650 the associated action for the associated (target) application. Thus, the received input may result in different commands being given to different applications based on the number of (potentially) used objects.

[0125] Input may be received (640) by detecting (642) a gesture, whereby a determined associated action 650 is to perform (652) a command associated with the gesture, the command being for an application associated with the number of objects.

[0126] The input may also, or alternatively, be received (640) by detecting a position and distance 644 of the detected object and detecting movement of the detected object (646), whereby a determined associated action 650 is to act according to the detected movement (655), such as by selecting a command associated with the detected position if the detected distance is less than a threshold, where the command is for an application associated with the number of objects.

[0127] 7 shows an overview of components of a software module (or component) arrangement 700 according to some embodiments of the teachings of the present disclosure. The software module arrangement 700 is adapted for use in the user device 100 as taught in the present disclosure to enable the user device 100 to perform the method according to FIG. 3. The software module arrangement 700 comprises a software module 710 for detecting a group of objects F in the touchless input area 104-4, a software module 720 for determining a number of objects in the group of objects, a software module 730 for determining an application associated with the number of objects, a software module 740 for accepting an input based on the detected group of objects F, and a software module 750 for determining an action associated with the input for an application associated with the number of objects.

[0128] 8 illustrates components of an arrangement 800 including a circuit. The arrangement including the circuit is adapted for use in the user device 100 as taught in the present disclosure to enable the user device 100 to perform the method illustrated in FIG. 3. The arrangement 800 includes a circuit 810 for detecting a group of objects F in the touchless input area 104-4, a circuit 820 for determining a number of objects in the group of objects, a circuit 830 for determining an application associated with the number of objects, a circuit 840 for accepting an input based on the detected group of objects F, and a circuit 850 for determining an action associated with the input of the application associated with the number of objects.

[0129] Further details regarding possible use of touchless interfaces are provided below in connection with a commonly-filed patent application entitled "Computer, Software Module Arrangement, Circuit Arrangement, User Device and Method for Improved Enhanced User Interface," filed concurrently with this application, with the understanding that corresponding circuits and software modules for the features described below (and features described in this disclosure) are deemed to be included in the illustrated modules and circuits even if not explicitly shown.

[0130] As described in relation to the previous figures, the side sensor 104-3 is capable of detecting an object (such as a user's finger) F and its position L and distance D to the display. In order to provide an enhanced user interface that does not interfere with the ongoing execution and / or presentation of the currently running application, i.e., the top application, the inventors have realized and devised a very intuitive menu interaction system that allows to provide many, almost infinite, control options without disturbing the top application. Such a menu interaction system is very beneficial for executing and / or controlling background applications and / or for providing system, general, or specific control for the user device 100. In addition, a number of objects are associated with an application. Also, by such association, multiple background applications can be controlled in addition to the top application. For example, based on the number of objects, using a single object controls the top application and using multiple fingers controls (different) background applications. Some examples include when a user wants to control volume settings or other controls for a music player application running in the background while playing a car driving game, handle incoming calls without interrupting the car driving game, respond quickly to messages, and / or control connected devices (such as a media device (TV) or other smart home devices).

[0131] The menu system can be utilized to execute and ultimately start another top application without interfering with the current top application by treating the second top application as a background application.

[0132] Some examples are when a user desires to select a particular music file to be played for a music player application running in the background while playing a car driving game (i.e., looking through a music library rather than skipping to the next song), set up and handle calls to particular contacts without interrupting the car driving game, and / or switch connection settings without interrupting the car driving game or other applications that are running.

[0133] To achieve this, the user device is configured to detect that an object, such as a user's finger F, is within the touchless input area 104-4 by the controller 101 receiving data indicative of this from the side sensor 104-3. The controller 101 also determines which portion of the touchless input area 104-4 the object F is within. According to some embodiments, the controller is configured to determine that the object is within the touchless input area 104-4 by determining that the distance D is less than an initial threshold distance.

[0134] When an object is detected, the controller 101 is configured, according to some embodiments, to indicate at least one menu option associated with the portion within which the object F is located. According to some embodiments, the controller is also configured to indicate that the portion is associated with a menu or array of commands, for example by providing feedback, by indicating the corresponding portion on the display 104-1. The feedback may be visual (graphics and / or lights) and / or audio. The feedback may also indicate that the subsection S1 is associated with a menu, and the extent of the menu.

[0135] Regardless of the feedback, the controller 101 is configured to show at least one menu option associated with the part within which the object F is located. In this context, a menu option may be a control command, a menu traversal command (up / back / next,...), further menu options and / or applications, all of which will be referred to below as options.

[0136] According to some embodiments, options may be displayed when it is determined that object F has been moved closer to display 104-1. In one context, "closer" refers to a distance that is less than the distance of the object when the feedback is displayed. In one context, "closer" refers to a distance that is less than a first threshold distance. Note that these two contexts could conceivably be combined, where the distance in the first context (the distance of the object when the feedback is displayed) defines the threshold distance.

[0137] The options are preferably displayed in a manner that is substantially transparent so as not to obscure content displayed in the same location on display 104-1. They are also displayed as transparent by the controller treating any input received within a display area overlapped by the graphical representation of the option as input for the underlying content, i.e., for the current top application.

[0138] Note that the options do not have to be displayed; the display of such options can be user-configurable, and a user can simply learn internally where the options are located without having to see the options displayed.

[0139] It should also be noted that the options may alternatively or additionally be presented with a sound that provides an indication of the option.

[0140] According to some embodiments, the options are simply displayed as an indication to guide the user to the location of the option, which is useful in situations where the user has the order of the options memorized, but perhaps not the exact location. Each option may be associated with a sub-portion.

[0141] According to some embodiments, the controller 101 is further configured to indicate which option is currently selected when a selection is made, i.e., which option the user is currently considered to be hovering over. The marking can be done by highlighting the option, changing the size of the option, changing the color of the option, changing the symbol of the option, and / or any combination thereof. The controller is further configured to determine that the object F has moved in a direction parallel to the display 104-1 (i.e., up or down in the plane of the display) and, in response thereto, indicate another option that is currently selectable.

[0142] The controller 101 is further configured to determine that the object F is to be moved closer to the display 104-1, thereby receiving a selection of an option at the object's location, and in response thereto, performing an action associated with the currently selectable option.

[0143] The associated action varies depending on the type of choice: for example, for a choice that is a menu traversal choice, the action is to perform a menu traversal (up / down / ...), for a choice that is a command, the action is to execute the command, and for a choice that corresponds to a deeper menu level, the action is to open a deeper or further menu level, thereby displaying at least one other choice. The other choice can be displayed in addition to and / or instead of the previous choice.

[0144] The controller 101 may also be configured to determine when an object is moved away from the display and in response return to a higher menu level, thus allowing traversal of a menu structure of options without substantially interfering with a running top application.

[0145] According to some embodiments, the controller 101 detects that the object has been moved closer to the display and, in response, performs an action associated with the selected option, which is to execute a command, in this example, the command to increase the volume setting for the user device 100.

[0146] In the example given in this disclosure, the volume is increased by pressing a virtual touchless button. However, other variations on such increase / decrease commands exist. One example is to select a function option (by moving towards the display) and then move a finger up to increase the associated function or down to decrease it. Examples of functions can be associated with volume, brightness, scrolling, toggle on / off switches, and adjustment settings to name a few.

[0147] Once an action has been performed, the controller 101 may be configured to continue displaying the currently displayed option if it is detected that the finger has moved slightly (to the next distance interval) away from the display, allowing further selection of the option. Alternatively or additionally, the controller is configured to stop displaying all options.

[0148] The controller 101 is also configured to stop displaying all options if it is determined that the object has moved away from the display, and no more options are displayed, however the effects of any commands that have been executed are still applicable.

[0149] According to some embodiments, the controller 101 is configured to determine that the object F has moved away from the display when the object is at a distance above an initial threshold distance.

[0150] According to some embodiments, the controller 101 is configured to determine that the object F has moved away from the display when the object F is no longer detectable.

[0151] According to some embodiments, the controller 101 is further configured to track the object F to ensure that commands are only activated when the object is approaching the display 104-1. This safeguard feature prevents the user from accidentally activating an action / command due to a change in grip or another object moving closer. Such a situation can easily arise when operating the user device 100 while sitting in a moving vehicle such as a train bogie or a bus.

[0152] A threshold range is viewed as a range between, and possibly including, multiple thresholds associated with the distance(s) in question.

[0153] Note that although the illustrations of this disclosure show three options (at one time), any number of options may be displayed, depending on the number of options available, the size of the subsection, the size of display 104-1, and the size of the indication.

[0154] 6, according to some embodiments, the controller 101 is configured to indicate (635) that at least one option is available for selection when object F is detected. This may be done by indicating a range of a menu structure and / or by displaying the at least one option, or rather by displaying a graphical representation of the at least one option.

[0155] The option that is in a position that corresponds to the object (such as corresponding to the position of the tip of the finger) is considered to be the selectable option. According to some embodiments, the controller 101 is configured to indicate which options are currently selectable.

[0156] As mentioned above, the controller 101 is configured to detect 646 movement of the object F and act 655 accordingly.

[0157] When movement is detected towards the user device 100, such as when the distance to the object falls below a threshold, the controller 101 is configured to operate by performing or implementing 660 an action associated with the currently selectable option, i.e., the option displayed in a position corresponding to where the distance to the object falls below the threshold distance.

[0158] If the selection is associated with a command, the controller 101 is configured to execute 662 such command. According to some embodiments, the controller is further configured to accept further input regarding the command and then execute based on the further input (e.g., move up to increase, move down to decrease, etc., as described above).

[0159] If the option is associated with another option, such as to display another option in a menu structure, the controller 101 is configured to display (664) the other option.

[0160] If the selection is associated with an application to be started, the controller 101 is configured to start (666) that application.

[0161] If the selection is associated with a data object, the controller is configured to select (668) the data object and possibly execute an associated command. One example of a data object might be a contact, and the associated action might be to initiate communication with the contact.

[0162] This allows for providing a way to traverse menu structures and / or complete user interface structures, where alternative options may be displayed, commands and functions may be executed, applications may be started, data objects may be selected, etc.

[0163] When movement is detected along or parallel to the user device 100, the controller 101 is configured to determine (670) new selectable options that correspond to the new position of the object, i.e., shift the options.

[0164] When motion is detected away from the user device 100, the controller 101 is configured to stop displaying (680) the at least one option, i.e., cancel the at least one option.

[0165] Other variations and areas of alternatives are possible for the various features as discussed above, some of which are discussed below, and it should be noted that they may be combined in whole, in part, or each as optional and compatible with the embodiments discussed above.

[0166] FIG. 9 shows a schematic diagram of a computer readable medium 120 carrying computer instructions 121 which, when loaded into and executed by a controller of the user equipment 100, enable the user equipment 100 to implement the present invention.

[0167] The computer readable medium 120 may be tangible, such as a hard drive or flash memory, e.g. a USB memory stick or a cloud server, or alternatively, the computer readable medium 120 may be intangible, such as a signal carrying computer instructions that allows the computer instructions to be downloaded over a network connection, such as an Internet connection.

[0168] In the example of FIG. 9, computer readable medium 120 is shown as a computer disk 120 carrying computer readable computer instructions 121 and inserted into computer disk reader 122. Computer disk reader 122 may be part of cloud server 123 or part of another server, or computer disk reader may be connected to cloud server 123 or another server. Cloud server 123 may be part of the Internet or at least connected to the Internet. Alternatively, cloud server 123 may be connected via a proprietary or dedicated connection. According to some exemplary embodiments, computer instructions are stored on remote server 123 and downloaded to memory 102 of user device 100 for execution by controller 101.

[0169] A computer disk reader 122 may also, or alternatively, be connected to (or possibly inserted into) the user device 100 for transferring computer-readable computer instructions 121 to the controller 101 of the user device 100 (possibly via the memory 102 of the user device 100).

[0170] 9 illustrates both the situation where a user device 100 receives the computer readable computer instructions 121 over a server connection, and also the situation where another user device 100 receives the computer readable computer instructions 121 over a wired interface, allowing the computer readable computer instructions 121 to be downloaded to the user device 100, thereby enabling the user device 100 to operate in accordance with and implement the invention disclosed in this disclosure.

Claims

1. A user device (100) comprising at least one side sensor (104-3) and a controller (101), The side sensor (104-3) is a radar sensor or a capacitive proximity sensor and is configured to accept touchless user input, thereby providing a touchless input area (104-4), and the controller (101) Detecting a group of objects (F) in the touchless input area (104-4); determining a number of objects in the group of objects; determining an application associated with the number of objects; Accepting input based on the group of detected objects (F); determining an action associated with the input for the application associated with the number of objects; The user device is configured to:

2. 2. The user device (100) of claim 1, wherein the controller is configured to determine a first number of objects and associate the first number of objects with a first application, and to determine a second number of objects and associate the second number of objects with a second application.

3. 3. A user device (100) according to claim 1 or 2, wherein the at least one side sensor (104-3) is configured to distinguish between different types of objects, and the controller (101) is further configured to: determining a type of object within the group of objects; determining the application associated with the number of objects and the type of objects; The user device is further configured.

4. 4. The user equipment (100) of claim 3, wherein the controller (101) determining an order of the two types of objects within the group of objects; determining the application associated with the number of objects, the type of objects, and the order of the objects; The user device is configured to:

5. The user device (100) of any one of claims 1 to 4, wherein at least one object in the group of detected objects is a finger.

6. The user device (100) of any one of claims 1 to 5, wherein at least one object in the group of detected objects is a stylus or pen.

7. 7. A user device (100) as claimed in any one of claims 1 to 6, wherein the controller is further configured to determine a number of objects in the group of detected objects by determining a location area of ​​the group of detected objects.

8. 8. The user device (100) of claim 1, wherein the controller is further configured to determine the number of objects in the group of detected objects by detecting at least one individual object.

9. A user equipment (100) according to any one of claims 1 to 8, wherein the controller is accepting said input by detecting a gesture (G); determining the associated action by executing a command associated with the detected gesture for the application associated with the number of objects; The user device is further configured.

10. A user equipment (100) according to any one of claims 1 to 9, wherein the controller is Detecting the position and distance of the group of detected objects (F); Detecting the movement of the group of detected objects (F). and accepting said input by and operating for the application associated with the number of objects according to the detected movement. determining the associated action by The user device is further configured.

11. 11. The user equipment (100) of claim 10, wherein the controller: performing an action associated with an option displayed at a position corresponding to the object when the motion is detected to be toward the user device (100); determining a new option corresponding to a new position of the object (F) if the movement is detected along the user device (100); canceling at least one selection if the motion is detected to be away from the user device (100); Execute a command if the option is associated with such a command and configured to operate for the application associated with the number of objects in response to the detected movement.

12. A user device (100) as described in any one of claims 1 to 11, wherein the user device (100) comprises a display (104-1), and at least one of the at least one side sensor (104-3) is arranged adjacent to the display (104-1), and the side sensor (104-3) is configured to accept the touchless user input at a side portion of the display (104-1), thereby providing the touchless input area (104-4) at the side portion of the display (104-1).

13. A user device (100) as described in any one of claims 1 to 12, wherein the user device (100) comprises an input device (100-2), and at least one of the at least one side sensor (104-3) is arranged adjacent to the input device (100-2), and the side sensor (104-3) is configured to accept the touchless user input at a side portion of the input device (100-2), thereby providing the touchless input area (104-4) at the side portion of the input device (100-2).

14. A user device (100) according to any one of claims 1 to 13, wherein the user device (100) comprises the side sensor (104-3) by being operatively connected to the user device (100).

15. 15. A user device (100) according to any one of claims 1 to 14, wherein the user device (100) is a smartphone, a smartwatch or a tablet computer.

16. A user device (100) according to any one of claims 1 to 15, wherein the user device (100) is a computer (100-1) configured to be operatively connected to a keyboard (100-2), a cursor control device (100-2), and / or a display (100-2, 104-1).

17. A method for use in a user device (100) comprising at least one side sensor (104-3), the side sensor being a radar sensor or a capacitive proximity sensor, configured to accept touchless user input, thereby providing a touchless input area (104-4), comprising: The method comprises: Detecting (610) a group of objects (F) within the touchless input area (104-4); determining (620) a number of objects in the group of objects; determining (630) an application associated with the number of objects; accepting (640) an input based on the group of detected objects (F); determining (650) an action associated with the input for the application associated with the number of objects; The method according to claim 1,

18. A computer-readable medium (120) carrying computer instructions (121) which, when loaded into and executed by a controller (101) of a user device (100), enable the user device (100) to perform the method of claim 16.

19. 1. A computer program (700) for a user device (100) comprising a display (104-1) and at least one side sensor (104-3), being a radar sensor or a capacitive proximity sensor, configured to accept touchless user input at a side portion of said display (104-1), thereby providing a touchless input area (104-4), said computer program (700) comprising: a software module (710) for detecting a group of objects (F) within said touchless input area (104-4); a software module (720) for determining a number of objects in the group of objects; a software module (730) for determining an application associated with the number of objects; a software module (740) for accepting input based on the group of detected objects (F); a software module (750) for determining an action associated with the input for the application associated with the number of objects; 13. A computer program comprising:

20. 1. A device (800) adapted for use in a user device (100) comprising a display (104-1) and at least one side sensor (104-3), which is a radar sensor or a capacitive proximity sensor, configured to accept touchless user input at a side portion of said display (104-1), thereby providing a touchless input area (104-4), said device (800) comprising: a circuit (810) for detecting a group of objects (F) within said touchless input area (104-4); a circuit (820) for determining a number of objects in the group of objects; a circuit for determining an application associated with the number of objects; a circuit (840) for accepting input based on a group of the detected objects (F); a circuit for determining an action associated with the input for the application associated with the number of objects; An apparatus having the above configuration.

Images