In-Air Control Area

Abstract

Aspects of the present disclosure relate to in-air control areas. Communication between an electronic pen and a device can be established. Two or more in-air control areas can be defined above the device, each designating a set of functions that can be performed by the electronic pen. A determination can be made whether the electronic pen is within a first of the two or more in-air control areas, the first in-air control area designating a first set of functions that can be performed by the electronic pen. At least one function of the first set of functions can be executed in response to input received from the electronic pen within the first in-air control area.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to the field of computing, and more particularly to the in-air control area.

[0002] An electronic pen (i.e., smart pen or digital pen) is an input device that allows a user to interact with electronic devices such as tablets and mobile phones. Handwritten data generated by a user using an electronic pen can be digitized for use on the electronic device. Digital pens can contain internal electronic features that facilitate input interaction, memory storage, and data transmission capabilities. Summary of the Invention

[0003] Embodiments of the present disclosure are directed to methods, systems, and computer program products for operation of an in-air control area. Communication between an electronic pen and a device may be established. Two or more in-air control areas may be defined above the device, each designating a set of functions that may be performed by the electronic pen. A determination may be made that the electronic pen is within a first of the two or more in-air control areas, the first in-air control area designating a first set of functions that may be performed by the electronic pen. At least one function of the first set of functions may be executed in response to input received from the electronic pen within the first in-air control area.

[0004] The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure. [Brief explanation of the drawings]

[0005] The drawings included in this disclosure are incorporated in and form a part of this specification. They illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings illustrate typical embodiments only and are not intended to limit the disclosure.

[0006] [Figure 1] FIG. 1 is a high-level block diagram illustrating an exemplary computer system and network environment that may be used to implement one or more of the methods, tools, modules, and any associated functionality described herein, according to embodiments of the present disclosure.

[0007] [Figure 2] FIG. 1 is a block diagram illustrating an exemplary computing environment in which exemplary embodiments of the present disclosure may be implemented.

[0008] [Figure 3] 1 is a diagram illustrating an exemplary in-air control region configuration implemented between a device and an electronic pen, according to an embodiment of the present disclosure.

[0009] [Figure 4] FIG. 1 is a flow diagram illustrating an exemplary method for operation of an in-air control region, according to an embodiment of the present disclosure.

[0010] While the embodiments described herein are amenable to various modifications and alternative forms, specific features thereof have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that the particular embodiments described are not to be taken in a limiting sense. Rather, the invention covers all modifications, equivalents, and alternatives falling within the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Aspects of the present disclosure relate generally to the field of computing, and more particularly to the in-air control area. The present disclosure is not necessarily limited to such applications, and various aspects of the present disclosure may be understood through a discussion of various examples used in this context.

[0012] An electronic pen (i.e., smart pen or digital pen) is an input device that allows a user to interact with electronic devices such as tablets and mobile phones. Handwritten data generated by a user using an electronic pen can be digitized on the electronic device. Digital pens can contain internal electronic features that facilitate input interaction, memory storage, and data transmission capabilities.

[0013] Touchscreen commands are typically issued between an electronic pen and an electronic device. For example, a user can use an electronic pen to write by hand on the touchscreen of an electronic device, and analog data associated with the handwriting can be converted to digital data. This may allow a user to write by hand on an electronic device in a manner similar to pen and paper. The electronic pen may be used for a variety of interface tasks, such as writing, mode switching, and scrolling.

[0014] In some cases, there may not be a suitable physical surface for the electronic pen to interface with (e.g., if the touchscreen is covered, obstructed, or otherwise unavailable). In these situations, in-air commands (e.g., mid-air commands, hover commands, etc.) may be issued on the electronic device using the electronic pen. These commands do not require direct physical contact between the electronic pen and the touchscreen. Rather, the commands are received in the air above the device. However, in-air command functionality is currently limited. For example, in-air commands typically do not take into account the distance between the electronic pen and the touchscreen of the electronic device. Furthermore, in-air commands do not change function based on the distance between the electronic pen and the touchscreen of the electronic device.

[0015] Aspects of the present disclosure relate to in-air control areas. Communication (e.g., pairing) between an electronic pen and a device can be established. Two or more in-air control areas can be defined above the device, each designating a set of functions that can be performed by the electronic pen. A determination can be made that the electronic pen is within a first of the two or more in-air control areas, the first in-air control area designating a first set of functions that can be performed by the electronic pen. At least one function of the first set of functions can be executed in response to input received from the electronic pen within the first in-air control area.

[0016] Aspects of the present disclosure provide various improvements to existing in-air command functionality. Aspects of the present disclosure target multiple in-air control areas above the device with different functions, thereby providing additional control options to the user. For example, a user may simply change the height (e.g., or other dimension) of their electronic pen relative to the device to perform different functions (e.g., using the same or similar input mechanism). Furthermore, because aspects of the present disclosure relate to visualizing various in-air control areas above the device (e.g., via extended reality (XR) and holographic projection technologies), the user may easily know which in-air control area they are currently interacting with, thereby improving the usability of the in-air command system.

[0017] Various aspects of the present disclosure are described through text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least overlaps in time, again depending on the technology involved.

[0018] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media"), collectively contained in one or more storage devices, that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as pits / lands formed on the major surface of a punch card or disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, should not be construed as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As one skilled in the art will appreciate, data is typically moved at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but the above does not qualify a storage device as transient because the data is not transient while it is stored.

[0019] Computing environment 100 includes an example environment for the execution of at least a portion of computer code involved in implementing the inventive methodology, such as in-air control region management 150. Computing environment 100 also includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes a set of processors 110 (including processing circuitry 120 and cache 121), a communications fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and block 150, as identified above), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage 124, and a set of Internet of Things (IoT) sensors 125), and a network module 115. Remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0020] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 130. As is well understood and in accordance with the art of computer technology, execution of a computer-implemented method may be distributed among multiple computers and / or among multiple locations. However, in this presentation of computing environment 100, to keep the presentation as simple as possible, the detailed discussion focuses on a single computer, specifically computer 101. Computer 101 may be located within a cloud, even though it is not depicted in the cloud of FIG. 1 . However, computer 101 is not required to reside within a cloud except to any extent that may be expressly indicated.

[0021] Processor set 110 includes one or more computer processors of any type now known or later developed. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple tailored integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of a processor set's cache may be located "off-chip." In some computing environments, processor set 110 may be designed for operations on qubits and for performing quantum computations.

[0022] Computer-readable program instructions are typically loaded onto the computer 101 to cause the processor set 110 of the computer 101 to perform a series of operational steps, thereby realizing a computer-implemented method, such that the instructions so executed instantiate the method specified in the flowcharts and / or descriptions of the computer-implemented methods contained herein (collectively referred to as the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media described below. The program instructions and associated data are accessed by processor set 110 to control and direct the execution of the methods of the present invention. In computing environment 100, at least some of the instructions for executing the methods of the present invention may be stored in block 150 in persistent storage 113.

[0023] Communications fabric 111 is the signal-conducting pathway that allows various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as those that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication pathways, such as fiber optic and / or wireless communication pathways, may be used.

[0024] Volatile memory 112 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located in a single package and internal to computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0025] Persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that stored data remains regardless of whether power is supplied to computer 101 and / or to persistent storage 113 directly. Persistent storage 113 may be read-only memory (ROM), but typically is at least a portion of persistent storage that allows data to be written, data to be erased, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems or an open-source Portable Operating System Interface-style operating system that uses a kernel. The code contained in block 150 typically includes at least some of the computer code involved in performing the methods of the present invention.

[0026] The peripheral device set 114 includes a set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth® connections, near field communication (NFC) connections, connections formed by cables (such as universal serial bus (USB)-type cables), insertion-type connections (e.g., Secure Digital (SD) cards), connections formed through local area communication networks, and even connections formed through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as display screens, speakers, microphones, wearable devices (e.g., goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, mixed reality (MR) headsets, and haptic devices. The storage 124 may be external storage such as an external hard drive or insertable storage such as an SD card. The storage 124 may be persistent and / or volatile. In some embodiments, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (e.g., computer 101 stores and manages a large database locally), this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 125 consists of sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0027] Network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers over WAN 102. Network module 115 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control function manages multiple different network hardware devices. Computer-readable program instructions for implementing the methods of the invention can be downloaded to computer 101 from an external computer or external storage device, typically via a network adapter card or network interface included in network module 115.

[0028] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances using any technology for communicating computer data now known or later developed. In some embodiments, a WAN may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include copper transmission cables, optical fiber transmissions, wireless transmissions, and computer hardware such as routers, firewalls, switches, gateway computers, and edge servers.

[0029] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and may take any of the forms described above in connection with computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from computer 101's network module 115 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, etc.

[0030] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0031] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer functionality, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of the computing resources of the public cloud 105 is performed by computer hardware and / or software in a cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers comprising a host physical machine set 142, which is the universe of physical computers within and / or available in the public cloud 105. A virtual computing environment (VCE) typically takes the form of virtual machines from a virtual machine set 143 and / or containers from a container set 144. It is understood that these VCEs may be stored as images and transferred among and between various hosts of physical machines either as images or after instantiation of the VCE. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that enables public cloud 105 to communicate over WAN 102.

[0032] Some further description of virtualized computing environments (VCEs) is now provided. A VCE can be stored as an "image." A new, active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows the existence of multiple isolated user space instances, called containers. These isolated user space instances typically behave as actual computers from the perspective of programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to the container; this feature is known as containerization.

[0033] A private cloud 106 is similar to a public cloud 105, except that the computing resources are available only for use by a single enterprise. While the private cloud 106 is shown in communication with the WAN 102, in other embodiments, the private cloud may be entirely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often each implemented by a different vendor. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0034] Referring now to Figure 2, a block diagram illustrates an exemplary computing environment 200 in which exemplary embodiments of the present disclosure may be implemented. The computing environment 200 includes an electronic device 205, an electronic pen 265, and a cross reality (XR) device 255. The electronic device 205, the electronic pen 265, and the XR device 255 may be remote from each other and may communicate via a network 250. The network 250 may be the same as or substantially similar to the WAN 102 described with respect to Figure 1. In an embodiment, the network 250 may be a local area network (LAN) or any other suitable network topology (e.g., peer-to-peer).

[0035] The electronic device 205 and the electronic pen 265 each include a processor 215, 285 and a memory 220, 290, respectively. The processors 215 and 285 may be the same as or substantially similar to the processor set 110 described with respect to Figure 1. The memories 220 and 290 may be the same as or substantially similar to the volatile memory 112 and / or persistent storage 113 described with respect to Figure 1. The functions described herein may be performed by one or more processors 215 of the electronic device, one or more processors 285 of the electronic pen, and / or one or more processors of a remote computing system (e.g., a server, not shown).

[0036] The in-air control region management application 225 of the electronic device 205 may be configured to receive in-air input commands from the electronic pen 265. An “in-air input command,” “in-air command,” or “in-air control” is a command / control (e.g., a mid-air command, a hover command, etc.) received in the air within three-dimensional (3-D) space in the device's environment. Although reference may be made to an in-air control region / command being defined / received “above the surface of the device,” the in-air control region / command may be defined / received in any suitable spatial orientation relative to the device (e.g., adjacent to or below it). The in-air input command may be detected by one or more sensors of the participating device (e.g., the sensor 270 of the electronic pen or the sensor 210 of the electronic device 205) and translated into a corresponding action or output. The in-air command executed from the electronic pen 265 may be processed based on the in-air control region defined above the electronic device 205. An "in-air control area" designates a 3-D area above the electronic device 105 where in-air command execution from the electronic pen 265 may be received (see, e.g., FIG. 2). Two or more in-air control areas may be defined above the electronic device 205. Each in-air control area may define a set of commands, functions, and / or permissions that may be performed by the electronic pen 265.

[0037] The sensor 210 of the electronic device 205 and / or the sensor 270 of the electronic pen 265 may be configured to track the position of the electronic pen 265 relative to the position of the electronic device 205. As such, during operation, the in-air control region within which the electronic pen 265 is positioned while performing a function (e.g., via the input function 280) may be determined such that the appropriate function is performed depending on the current in-air region within which the electronic pen 265 is positioned. Tracking of the position of the electronic pen 265 relative to the electronic device 205 may be continuous. That is, the sensors 210 and 270 may constantly track the position of the electronic pen 265 to determine whether the electronic pen is within an in-air control region of the electronic device 205, the particular in-air control region within which the electronic pen 265 is located, and whether the electronic pen 265 transitions between in-air control regions of the electronic device 205.

[0038] The sensors 210 and 270 can be light-based (e.g., cameras, optical sensors, infrared sensors, light detection and ranging (LIDAR) sensors, etc.), movement-based (e.g., accelerometers, gyroscopes, etc.), sound-based (e.g., ultrasonic sensors, microphones, etc.), touch-based (e.g., capacitive sensors, tactile sensors, etc.), and network-based (e.g., sensors implemented in an indoor positioning system (IPS) such as global positioning system (GPS) sensors, wireless network sensors (e.g., Wi-Fi or Bluetooth positioning systems), received signal strength indicator (RSSI) sensors, etc.). In an embodiment, two or more of the above-referenced sensor types may be collectively implemented and used to track the position of the electronic pen 265 relative to the in-air control area of the electronic device 205.

[0039] In embodiments, if a determination is made (e.g., based on sensor data collected from sensors 210 and 270) that the electronic pen will transition from a first in-air control area to a second in-air control area, the tactile actuator 275 of the electronic pen may be configured to provide tactile feedback regarding the in-air control area transition. This may be accomplished so that a user of the electronic pen 265 can be notified of any in-air control area changes that have occurred. This may enable the user to perform a function according to the in-air control area in which the electronic pen 265 is currently located. This may also enable the user to be notified of an in-air control area transition when it occurs (e.g., whether intentional or unintentional). However, in embodiments, any other suitable notification of an in-air control area transition may be accomplished. For example, a visual and / or audio indication (e.g., a message displayed on the electronic device 205 or a sound emitted from the electronic device 105) may be issued from the electronic device 205 and / or the electronic pen 265 to notify the user of the in-air control area transition.

[0040] In embodiments, the in-air control areas generated by the in-air control area management application 225 may be displayed to a user via an XR device 255. The XR device 255 may be a virtual reality (VR), augmented reality (AR), or mixed reality (MR) display device configured to display virtual visual data to a user, among other potential sensor data (e.g., haptic and auditory data). The XR device 255 may be implemented using any suitable VR, AR, and / or MR technology, including tracking sensors, one or more virtual boundaries, indoor positioning system capabilities, a head-mounted display (HMD), graphics processing capabilities, data transmission capabilities, etc. The XR display 260 of the XR device 255 may be configured to display the geometry (e.g., 3-D area) of each in-air control area above the electronic device 205. This may improve the usability of the system by allowing a user to visualize the in-air control area with which they are currently interacting. The display of the in-air regions via the XR display 260 may include a virtual border / perimeter indicating the boundaries of each in-air control region.

[0041] In embodiments, as opposed to or in addition to viewing the in-air boundary control areas via XR display 260, holographic display 230 of electronic device 205 may be configured to project a hologram showing the geometry of each in-air control area above electronic device 205. The holographic display uses light created by, for example, a laser to create a 3-D image in space (e.g., in the air above device 205). This may allow a user to view a virtual boundary / perimeter showing the boundaries of each in-air control area without requiring the user to wear an XR device (e.g., XR device 255).

[0042] The in-air control area of the in-air control area management application 225 may be defined in any suitable manner. In an embodiment, a user may manually define the in-air control area. For example, a user may manually trace, specify, or otherwise map the in-air control area they desire. Tracing the boundary of the in-air control area may involve the user drawing the geometry of the in-air control area (e.g., in the air above the device or on the surface of the device). The traced boundary may then be defined as a 3-D in-air control area based on the size and shape of the boundary drawn by the user. In an embodiment, the traced boundary may be detected by the sensor 210 of the electronic device 205 or the sensor 270 of the electronic pen 265 (e.g., if the electronic pen 265 is performing the tracing).

[0043] However, manual definition of an in-air control area may be completed in any suitable manner. For example, the definition may be completed within a software application (e.g., in-air control area management application 225). For example, a user may define dimensions (e.g., within input fields of the software application) specifying the geometry of the in-air control area they desire. As a specific example, for a rectangular or square in-air control area geometry, the user may indicate the desired height, width, and / or length of the rectangular in-air control area and the starting distance from electronic device 205 from which the rectangular in-air control area is to be projected. As another specific example, for a circular geometry, the user may indicate the circumference, radius, diameter, and / or height of the circular in-air control area and the starting distance from electronic device 205 from which the circular in-air control area is to be projected. Manual definition may be completed for each of multiple in-air control areas.

[0044] In embodiments, the in-air control areas may be automatically generated. For example, in some embodiments, the automatic generation of the in-air control areas may be completed based on the accuracy / precision of the sensors. As an example, if the sensor 210 and / or the sensor 270 can only accurately distinguish the distance between the device 205 and the electronic pen 265 to within 5 centimeters (cm) (e.g., in a height direction above the surface of the electronic device), the minimum distance between the in-air control areas may be set to at least 5 cm. As another example, the in-air control areas may be automatically generated based on previously captured in-air command data. For example, the distance from which the user has previously issued in-air commands may be taken as input data by a machine learning (ML) / artificial intelligence (AI) algorithm, and the ML / AI algorithm may output the geometry of the in-air control area based on the data of the previously issued in-air commands. Any suitable AI / ML algorithm may be used to automatically generate the in-air control region, including, but not limited to, supervised learning (e.g., support vector machines), unsupervised learning (e.g., clustering), reinforcement learning (e.g., Markov decision-based models), and / or deep learning (e.g., artificial neural networks).

[0045] In embodiments, the definition of the placement and geometry of the in-air control area may be completed partially manually and partially automatically. For example, two dimensions (e.g., length and width) of the in-air control area may be defined manually (e.g., based on user input), while a third dimension (e.g., height) of the in-air control area may be defined automatically (e.g., based on sensor accuracy). Similarly, a first in-air control area may be manually defined (e.g., traced by a user), while a second in-air control area may be automatically generated by an ML / AI algorithm (e.g., deep learning). Aspects of the present disclosure recognize that the definition of the geometry and placement of the in-air control area may be completed in any suitable manner and is not limited to those described.

[0046] In an embodiment, commands, functions, and / or permissions associated with each defined in-air control area may be defined. The definition of the commands, functions, and / or permissions associated with each in-air control area may be completed manually or automatically. For example, a user may specify within the in-air control area management application 225 that a first set of commands, functions, or permissions may be performed within a first in-air control area and a second set of commands, functions, or permissions may be performed within a second in-air control area. As another example, the commands, functions, and / or permissions associated with each in-air control area may be automatically defined based on historical in-air command data via ML / AI algorithms. For example, historical data capturing the type of commands, type of functions, and / or permissions completed for in-air commands previously issued by the electronic pen 265 at a particular distance / position from the electronic device 205 may be captured and used to define / assign the commands, functions, and / or permissions for each in-air control area.

[0047] Any suitable commands, functions, and / or permissions may be mapped to the in-air control areas. As one example, a zoom-in command may be permitted only within a first in-air control area, while a zoom-out command may be permitted only within a second in-air control area. As another example, read permission may be granted only within a first in-air control area, and read / write permission may be granted only within a second in-air control area. As another example, a first function (e.g., draw) may be completed only within a first in-air control area, while a second function (e.g., erase) may be completed only within a second in-air control area. In embodiments, a first set of commands, functions, and / or permissions may be manually assigned to each in-air control area, and a second set of commands, functions, and / or permissions may be automatically assigned to each in-air control area. In embodiments, input functions 280 (e.g., buttons, dials, switches, etc.) of the electronic pen 265 may be mapped to specific functions within each in-air control area. For example, a first button on the electronic pen 265 may perform a first function within a first in-air control area, while a first button on the electronic pen 265 may perform a second function within a second in-air control area.

[0048] In embodiments, multiple devices (e.g., additional electronic devices, not shown) may each have a defined in-air control area and a corresponding set of commands, functions, and / or permissions mapped to each in-air control area. In these embodiments, corresponding sensors may be configured to track the position of electronic pen 265 so that the appropriate functions may be performed according to the in-air control area within which electronic pen 265 is determined to be. In embodiments, if the in-air control areas of multiple respective devices overlap, the functions of each device associated with each in-air control area may be executed simultaneously.

[0049] It should be noted that Figure 2 is intended to illustrate representative major components of exemplary computing environment 200. However, in some embodiments, individual components may be more or less complex than depicted in Figure 2, components other than or in addition to those depicted in Figure 2 may be present, and the number, type, and configuration of such components may vary. For example, while Figure 2 illustrates in-air control region management application 225 as stored in memory 220 of electronic device 205, in embodiments, in-air control region management application 225 may be stored remotely on a server. In these embodiments, data captured by in-air control region management application 225 may be processed remotely by a remote server.

[0050] Referring now to FIG. 3 , a diagram illustrating an exemplary in-air control area configuration is shown, according to an embodiment of the present disclosure. As shown in FIG. 3 , an in-air control area 340 is displayed above a device 305 (e.g., a computer 101 or an electronic device 205). The in-air control area 340 includes a first in-air control area 310 closest to the device 305, a third in-air control area 320 farthest from the device 305, and a second in-air control area 315 between the first in-air control area 310 and the second in-air control area 320. The first in-air control area 310 and the second in-air control area are separated by a first imaginary boundary 325. The second in-air control area 315 and the third in-air control area 320 are separated by a second imaginary boundary 330. The endpoint of the third in-air control area 320 is designated by a third imaginary boundary 335.

[0051] Each in-air control area 340 may be mapped to a set of commands, functions, and / or permissions that can be completed within the respective in-air control area. Execution of the defined set of commands, functions, and / or permissions for each in-air control area may be completed by the electronic pen 350. The location of the electronic pen 350 (e.g., the tip of the electronic pen) relative to the in-air control area 340 may be continuously determined (e.g., via sensors 210 and / or 270) so that the appropriate functions, commands, and / or permissions can be performed / enforced. In embodiments, an input function 355 (e.g., shown as a button) of the electronic pen 350 may be used to input commands from the electronic pen 350 within each in-air control area. In embodiments, the function of an input command performed by the electronic pen 350 may differ based on the in-air control area the electronic pen 350 is within. For example, a first input button of the electronic pen 350 implemented within a first in-air control area may provide a first function (e.g., draw), while a first input button of the electronic pen 350 implemented within a second in-air control area may provide a second function (e.g., erase).

[0052] As described with respect to FIG. 2 , the in-air control area 340 may be visually displayed by an XR device worn by the user and / or via a holographic projection above the device 305. This may allow the user to easily know which in-air control area they are currently interacting with. Additionally, feedback (e.g., haptic, auditory, and / or visual feedback) may be presented to the user when entering a new in-air control area with the electronic pen 350 (e.g., transitioning from the first in-air control area 310 to the second in-air control area 315, or initially entering the third in-air control area 320). Providing the user with sensory data indicating which in-air control area they are currently interacting with improves the usability of the system.

[0053] Although the in-air control area 340 in FIG. 3 is shown as a rectangle, in embodiments, any suitable geometry for the in-air control area may be implemented (e.g., circular, oval, hexagonal, etc.). The dimensions of the in-air control areas may vary and may be defined manually or automatically. In embodiments, the geometry of each in-air control area may vary. For example, the first in-air control area 310 may be rectangular, the second in-air control area 315 may be circular, and the third in-air control area 320 may be triangular. In embodiments, the size of the in-air control areas may vary based on distance. For example, the first in-air control area 310 may be the smallest (e.g., the apex of a pyramid), the second in-air control area may be of intermediate size (e.g., the middle of the pyramid), and the third in-air control area may be the largest (e.g., the base of the pyramid). Additionally, the placement of the in-air control areas (e.g., starting distance from the device 305) may vary. For example, although the first in-air control region 310 is shown as starting immediately above the device 305, in embodiments, the first in-air control region 310 may start offset a particular distance (e.g., 10 cm) above the device.

[0054] Although reference has been made to tracking the position of the electronic pen 350 relative to the in-air control areas, aspects of the present disclosure recognize that a first portion (e.g., tip) of the electronic pen 350 may be within a first in-air control area and a second portion (e.g., end) of the electronic pen 350 may be within a second in-air control area. Aspects of the present disclosure recognize that tracking may be performed on a single component (e.g., tip) of the electronic pen 350 such that precise positioning / commanding may be achieved.

[0055] 4, a flow diagram illustrating an exemplary method 400 for operation of an in-air control region is shown, according to an embodiment of the present disclosure. One or more operations of method 400 may be completed by one or more computing devices (e.g., computer 101, electronic device 205, electronic pen 265, electronic device 305, electronic pen 350).

[0056] Method 400 begins at operation 405, where in-air control areas and associated commands, functions, and / or permissions for each in-air control area are defined for a device (e.g., computer 101, electronic device 205, device 305). Defining the placement / geometry of the in-air control areas may be completed in the same or similar manner (e.g., manually or automatically) as described with respect to Figure 2. Defining / mapping the commands, functions, and / or permissions associated with each respective in-air control area may likewise be completed in the same or substantially similar manner (e.g., manually or automatically) as described with respect to Figure 2.

[0057] Communication (e.g., communication coupling or pairing) between the electronic pen (e.g., electronic pen 265, electronic pen 350) and the device is established. This is shown in operation 410. Communication between the electronic pen and the device may be completed in any suitable manner, such as over a wireless network (e.g., Wi-Fi, Bluetooth, etc.).

[0058] The position of the electronic pen relative to the device is determined. This is shown in operation 415. Tracking the position of the electronic pen relative to the device may be completed in the same or substantially similar manner as described with respect to Figure 2. For example, tracking the position of the electronic pen relative to the device may be completed using one or more sensors (e.g., motion-based sensors, light-based sensors, network-based sensors, etc.).

[0059] A determination is made whether the electronic pen is within the in-air control region of the device. This is shown in operation 420. If a determination is made that the electronic pen is not within the in-air control region of the device (“No” at operation 420), method 400 may return to operation 415, where the position of the electronic pen relative to the device may continue to be determined. In embodiments, tracking of the electronic pen may begin only if the electronic pen is within a threshold range (e.g., a minimum distance) of the electronic device. In embodiments, tracking of the position of the electronic pen relative to the device may be performed intermittently, continuously, periodically, or at any other suitable interval.

[0060] If a determination is made that the electronic pen is within the in-air control area of the device (“Yes” at operation 420), the commands, functions, and / or permissions associated with the in-air control area the electronic pen is within are executed / enforced according to the received input. This is shown at operation 425.

[0061] A determination is made whether the electronic pen changes (e.g., transitions) the in-air control region. This is shown in operation 430. The determination whether the electronic pen changes the in-air control region may be completed by tracking the position of the electronic pen relative to the device. If a determination is made that the electronic pen did not change the in-air control region ("No" at operation 430), the position of the electronic pen relative to the device may continue to be tracked to detect any in-air control region transitions that may occur.

[0062] If a determination is made that the electronic pen has changed the in-air control area ("Yes" at operation 430), feedback (e.g., tactile, auditory, and / or visual feedback) indicating the transition may be issued to the user of the electronic pen. This is shown at operation 435. The feedback may be issued from the electronic pen and / or from the device. In an embodiment, a haptic actuator in the electronic pen provides tactile feedback to the user upon any in-air control area transitions.

[0063] The commands, functions, and / or permissions associated with the new in-air control region may then be executed according to the received input of the electronic pen, as shown in operation 440.

[0064] The foregoing operations may be completed in any order and are not limited to those described. Furthermore, some, all, or none of the foregoing operations may be completed while still remaining within the scope of the present disclosure.

[0065] As described in more detail herein, some or all of the operations of some of the method embodiments described herein may be performed in an alternative order, or not performed at all; further, multiple operations may occur simultaneously or as part of a larger process.

[0066] The present disclosure may be a system, a method, and / or a computer program product, which may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present disclosure.

[0067] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as being a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0068] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0069] The computer-readable program instructions for carrying out the operations of the present disclosure may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages like Smalltalk, C++, or the like, and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to individually configure the electronic circuit to perform aspects of the present disclosure.

[0070] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine, whereby the instructions executing on the processor of the computer or other programmable data processing apparatus form means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium capable of instructing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, whereby the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture having instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0072] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0073] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be implemented as a single step, executed concurrently, substantially concurrently, partially, or fully in an overlapping manner, or the blocks may possibly be executed in reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0074] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit various embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "includes" and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In the preceding detailed description of exemplary embodiments of various embodiments, reference has been made to the accompanying drawings, which form a part hereof, in which like numerals represent like elements, and in which are shown, by way of illustration, specific exemplary embodiments in which various embodiments may be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the embodiments, but other embodiments may be used, and logical, mechanical, electronic, and other changes may be made without departing from the scope of the various embodiments. In the foregoing description, numerous specific details have been set forth to provide a thorough understanding of various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail so as not to obscure the embodiments.

[0075] Different instances of the term "embodiment" as used herein do not necessarily refer to the same embodiment, although they may. Any data and data structures shown or described herein are examples only; other embodiments may use different amounts of data, types of data, fields, numbers and types of fields, field names, numbers and types of rows, records, entries, or organizations of data. Additionally, any data may be combined with logic, such that a separate data structure may not be necessary. Therefore, the preceding detailed description should not be taken in a limiting sense.

[0076] The description of various embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements over commercially available technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0077] While the present disclosure has been described with respect to specific embodiments, it is anticipated that variations and modifications thereof will become apparent to those skilled in the art, and it is therefore intended that the following claims be interpreted to cover all such variations and modifications that fall within the scope of the present disclosure.

Claims

1. one or more processors; and When executed by the one or more processors, the one or more processors associated with the electronic pen and device: Steps to establish communication between the pen and the device: defining two or more in-air control areas above the device, each in-air control area specifying a set of functions that can be performed by the electronic pen; determining that the electronic pen is within a first in-air control area of the two or more in-air control areas, the first in-air control area specifying a first set of functions that can be performed by the electronic pen; and performing at least one function of the first set of functions in response to input received from the electronic pen within the first in-air control area. One or more computer-readable storage media collectively storing program instructions configured to cause a method having A system comprising:

2. The method implemented by the one or more processors: determining that the electronic pen has transitioned from the first in-air control area of the two or more in-air control areas to a second in-air control area of the two or more in-air control areas, the second in-air control area specifying a second set of functions that can be performed by the electronic pen; and generating tactile feedback by a tactile actuator of the electronic pen to notify a user of the electronic pen that the transition has occurred. The system of claim 1 further comprising:

3. The method implemented by the one or more processors: performing at least one function of the second set of functions in response to input received from the electronic pen within the second in-air control area. The system of claim 2 further comprising:

4. 10. The system of claim 1, wherein the two or more in-air control regions are displayed as virtual boundaries by a cross reality (XR) device worn by a user.

5. The system of claim 1 , wherein the two or more in-air control areas are displayed as virtual boundaries by a holographic display of the device.

6. The system of claim 1 , wherein the two or more in-air control regions are manually defined by a user.

7. The system of claim 6 , wherein the two or more in-air control regions are manually defined by the user via in-air tracing.

8. The system of claim 1 , wherein the two or more in-air control regions are automatically defined.

9. 2. The system of claim 1, wherein at least one in-air control area of the two or more in-air control areas is manually defined and at least one in-air control area of the two or more in-air control areas is automatically defined.

10. 2. The system of claim 1, wherein the first in-air control area of the two or more in-air control areas has a first geometry and the second in-air control area of the two or more in-air control areas has a second geometry.

11. The system of claim 1 , wherein the first in-air control area is offset a distance above the device.

12. establishing communication between the electronic pen and the device; defining two or more in-air control areas above the device, each in-air control area specifying a set of functions that can be performed by the electronic pen; determining that the electronic pen is within a first in-air control area of the two or more in-air control areas, the first in-air control area specifying a first set of functions that can be performed by the electronic pen; and performing at least one function of the first set of functions in response to input received from the electronic pen within the first in-air control area. A method comprising:

13. determining that the electronic pen has transitioned from the first in-air control area of the two or more in-air control areas to a second in-air control area of the two or more in-air control areas, the second in-air control area specifying a second set of functions that can be performed by the electronic pen; and generating tactile feedback by a tactile actuator of the electronic pen to notify a user of the electronic pen that the transition has occurred. The method of claim 12 further comprising:

14. performing at least one function of the second set of functions in response to input received from the electronic pen within the second in-air control area. The method of claim 13 further comprising:

15. The method of claim 12 , wherein the two or more in-air control regions are displayed as virtual boundaries by a cross reality (XR) device worn by a user.

16. The method of claim 12 , wherein the two or more in-air control regions are displayed as virtual boundaries by a holographic display of the device.

17. one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the program instructions causing one or more processors to: a procedure for establishing communication between the electronic pen and the device; defining two or more in-air control areas above the device, each in-air control area specifying a set of functions that can be performed by the electronic pen; determining that the electronic pen is in a first in-air control area of the two or more in-air control areas, the first in-air control area specifying a first set of functions that can be performed by the electronic pen; and performing at least one function of the first set of functions in response to input received from the electronic pen within the first in-air control area.

1. A computer program product having instructions configured to cause the computer to perform a method comprising:

18. The method implemented by the one or more processors: determining that the electronic pen has transitioned from the first in-air control area of the two or more in-air control areas to a second in-air control area of the two or more in-air control areas, the second in-air control area specifying a second set of functions that can be performed by the electronic pen; and generating tactile feedback by a tactile actuator of the electronic pen to notify a user of the electronic pen that the transition has occurred.

20. The computer program product of claim 17, further comprising:

19. The method implemented by the one or more processors: performing at least one function of the second set of functions in response to input received from the electronic pen within the second in-air control area.

20. The computer program product of claim 18, further comprising:

20. 20. The computer program product of claim 17, wherein the two or more in-air control regions are displayed as virtual boundaries by a cross reality (XR) device worn by a user.

Images