Pen and input device for use in an augmented / virtual reality environment

Abstract

Pen device (110, 310(1-3), 410, 510, 610, 710) designed for use in an Augmented / Virtual Reality (AR / VR) environment (100), wherein the pen device (110, 310(1-3), 410, 510, 610, 710) comprises: a housing that includes: a first part (312(1-3), 412, 512, 612, 712); and a second part (314(1-3), 414, 514, 614, 714); one or more processors (210) arranged in the housing; and a communication module (250) arranged in the housing and controlled by the one or more processors (210), wherein the communication module (250) is configured to establish a wireless electronic communication channel between the pen device (110, 310(1-3), 410, 510, 610, 710) and at least one host computer device (140), wherein the first part (312(1-3), 412, 512, 612, 712) of the housing is substantially straight and designed to be held by a user (102, 602) with one hand (104, 504, 604) during use of the pin device (110, 310(1-3), 410, 510, 610, 710), wherein the first part (312(1-3), 412, 512, 612, 712) comprises: an input element (416) configured to generate control signals in response to activation by the hand (104, 504, 604) of the user (102, 602); and a tip at one end of the first part (312(1-3), 412, 512, 612, 712) which is designed for handling as an interface between the pen device (110, 310(1-3), 410, 510, 610, 710) and objects within the AR / VR environment (100), wherein the second part (314(1-3), 414, 514, 614, 714) of the case is not straight and is curved in such a way that it extends in three dimensions, and wherein the second part (314(1-3), 414, 514, 614, 714) of the housing has several emitters or sensors (719) designed to support tracking of the pen device (110, 310(1-3), 410, 510, 610, 710) in the three-dimensional space of the AR / VR environment (100).

Description

[0001] Virtual, mixed, or augmented reality can be linked to various applications that include immersive, highly visual, computer-simulated environments. These environments, commonly referred to as augmented reality (AR) or virtual reality (VR) environments, simulate a user's physical presence in a real or imagined world. A computer simulation of such environments can include computer-generated images that can be displayed using a graphical display. This display can be a head-mounted display (HMD) that covers all or part of the user's field of vision.

[0002] A user can connect to the computer-simulated environment via a user interface device or peripheral. A common type of controller in many modern AR / VR systems is the pistol-grip controller, which, depending on the specific system, can typically be operated to track movement with three or six degrees of freedom (DOF). Once immersed in a computer-simulated AR / VR environment, a user can use the interface device to perform complex actions, including simulated movement, interaction with or manipulation of an object, and more.Despite their practicality, pistol-grip controllers in modern AR / VR systems are typically bulky, cumbersome, and unwieldy, and their weight and extensive tracking mechanisms, often housed in a cumbersome and protruding donut-shaped structure, can lead to user fatigue. While the pistol-grip shape can help minimize fatigue because users can typically hold pistol-grip objects for extended periods, this comes at the cost of coarse and imprecise movement and clumsy control. Therefore, there is a need for improved interface devices for interacting in virtual environments, especially when performing tasks that require a high degree of precision and fine control.

[0003] DE 10 2004 013 708 A1 discloses an interaction device for generating inputs that are transmitted via a transmitting device to a computer-based application. The interaction device is used for interaction with computer-based applications with spatially multidimensional image representations.

[0004] US 6,956,564 B1 discloses a portable computer in the form of a pen that responds to the user's movements through motion detection. The portable computer also has several buttons to facilitate operation.

[0005] US 2016 / 0062492A1 discloses a pin device with multiple transmitters whose signal can be detected on a grid of electrical connections. The position and inclination of the pin device can be determined based on the detected signals.

[0006] In some embodiments, a pen device can be configured for use in an augmented / virtual reality (AR / VR) environment, wherein the pen has a housing comprising: a first part and a second part; one or more processors arranged within the housing; and a communication module arranged within the housing and controlled by the one or more processors, the communication module being configured to establish a wireless electronic communication channel between the pen device and at least one host computer device. The first part of the housing can be substantially straight and configured to be held by hand by a user during use of the pen device.The first part can be an input element designed to generate control signals in response to activation by a user's hand, and may have a tip at one end designed for handling as an interface between the input device and objects within the AR / VR environment. The second part of the housing may be curved and extend in three dimensions (e.g., the x, y, and z directions of a Cartesian coordinate system), with a section of the second part curved lengthwise along a line collinear with the first part of the housing. In some cases, the second part of the housing may have multiple emitters or sensors designed to assist in tracking the pen device in the three-dimensional space of the AR / VR environment.

[0007] In certain embodiments, a section of the second part of the housing can form a polygonal ring (e.g., a hexagonal ring, a rhombic ring, a circular ring, a spherical ring, a hemispherical ring, an octagonal ring, etc.). In some cases, the second part of the housing can comprise a polygonal cross-section through at least a portion of the polygonal ring (e.g., a hexagonal cross-section, a rhombic cross-section, a circular cross-section, an octagonal cross-section, etc.). The second part of the housing can have multiple planar facets, with the multiple emitters or sensors being arranged on some or all of the multiple planar facets. Some or all of the multiple planar facets can be oriented to point in different directions in three-dimensional space.In some cases, some or all of the planar facets may be oriented such that at least four of the planar facets are visible from any point 360 degrees around the pen device. To illustrate this, if the first section of the housing (which may be designed to be held by a user in a pen-grip position) is essentially straight, any point 360 degrees perpendicular to the first section can be viewed such that at least four of the planar facets and / or emitters or sensors can be detected. As another, simplified example, an axis-wheel relationship can be given, where the first part of the pen device (the essentially straight part) may form the axis, and the 360 ​​degrees of viewing may represent the orientation of the wheel relative to the axis. In some cases, however, detection may be achieved at any point around the pen device (e.g.,(from any point on a sphere surrounding the input device). In some cases, the center of gravity of the pen device may be located at a transition between the second part and the first part. Furthermore, the pen device can, as in . Fig. As shown in Figure 6B, the transition between the second part and the first part must be balanced in the longitudinal and transverse directions (e.g. longitudinal and transverse).

[0008] In certain embodiments, an input device designed for use in an augmented / virtual reality (AR / VR) environment may comprise a housing comprising: a substantially straight first part designed to be handheld by a user during use of the input device; and a second part that is not straight and is curved to extend in three dimensions, the second part of the housing having multiple emitters or sensors designed to assist in tracking the input device in the three-dimensional space of the AR / VR environment. In some cases, a section of the second part may be curved longitudinally in the direction of a line collinear with the first part of the housing.In some embodiments, a section of the second part of the housing can form a hexagonal ring, with the housing having a hexagonal cross-section through at least a portion of the hexagonal ring. The second part of the housing can have multiple planar facets, with the multiple emitters or sensors being formed on the multiple planar facets. The multiple planar facets can each be oriented such that they point in different directions in three-dimensional space. In some cases, the multiple planar facets are oriented such that at least four of the planar facets are visible from any point 360 degrees around the input device. In certain embodiments, the input device can be a stylus device.The center of gravity of the input device can be located at the transition between the second and first parts, allowing the input device to be balanced without support in the space between the index finger and thumb of a user's hand when the user's hand is in an input device insertion position. In some implementations, the input device can be balanced longitudinally and transversely at a transition between the second and first parts.

[0009] In further embodiments, an input device designed for use in an augmented / virtual reality (AR / VR) environment may comprise a housing comprising: a first part; and a second part; wherein the first part of the housing may be designed to be held by hand by a user during use of the input device, wherein the second part of the housing may be non-straight and curved to extend in three dimensions, and has a section forming a hexagonal ring with a hexagonal cross-section, wherein the hexagonal ring may have a section curved longitudinally in the direction of a line collinear with the first part of the housing, wherein the hexagonal ring may have multiple planar facets, and wherein multiple emitters or sensors are arranged on the multiple planar facets.which are designed to support tracking of the input device in the three-dimensional space of the AR / VR environment. In some cases, the multiple planar facets can each be oriented so that they point in different directions within three-dimensional space, with each of the multiple planar facets being oriented such that at least four of the planar facets are visible from any point 360 degrees around the input device.

[0010] Aspects, features and advantages of embodiments of the present disclosure will become apparent from the following description of embodiments, which refers to the accompanying figures.

[0011] Fig. Figure 1 shows a user operating a pen device in an augmented / virtual reality environment, according to certain embodiments.

[0012] Fig. Figure 2 shows a simplified block diagram of an AR / VR system, according to certain embodiments.

[0013] Fig. Figure 3A shows a pen device designed for use in an AR / VR environment, according to certain embodiments.

[0014] Fig. Figure 3B shows a second implementation of a pen device designed for use in an AR / VR environment, according to certain embodiments.

[0015] The Fig. 3C-3E disclose a third implementation of a pen device designed for use in an AR / VR environment, according to certain embodiments.

[0016] Fig. Figure 4A shows aspects of a pen device for use in an AR / VR environment, according to certain embodiments.

[0017] Fig. Figure 4B shows aspects of a pen device for use in an AR / VR environment, according to certain embodiments.

[0018] Fig. Figure 4C shows aspects of a pen device for use in an AR / VR environment, according to certain embodiments.

[0019] Fig. Figure 5A shows a gripping position for connection with the pin device, according to certain embodiments.

[0020] Fig. Figure 5B shows a second gripping position for connection with the pin device, according to certain embodiments.

[0021] Fig. 6A shows aspects of certain ergonomic advantages of the pen device, according to certain embodiments.

[0022] Fig. Figure 6B shows further aspects of certain ergonomic advantages of the pen device, according to certain embodiments.

[0023] Fig. Section 6C shows further aspects of the ergonomic advantages of the pen device, according to certain embodiments.

[0024] Fig. Figure 7 shows aspects of the configuration and operating behavior of the sensor-emitter array of the pin device, according to certain embodiments.

[0025] Embodiments of this invention generally relate to control devices designed for use in AR / VR-based systems. More specifically, some embodiments relate to a pen device with a novel design architecture that exhibits improved ergonomic and tracking properties.

[0026] The examples and details given in the following description for illustrative purposes serve to facilitate understanding of the embodiments of the present invention. However, a person skilled in the art will recognize that certain embodiments can be realized without some of these details or with modifications or equivalents.

[0027] To facilitate a deeper and more comprehensive understanding of some aspects of the present disclosure, a non-limiting summary of certain embodiments is presented here. Many conventional AR / VR systems currently on the market commonly use pistol-grip controllers, which, however, suffer from ergonomic problems as described above. In summary, conventional pistol-grip controllers (i.e., input devices) are often cumbersome and, due to their bulky and unwieldy design, as well as the biomechanics of a user's hand in the pistol-grip configuration, which is generally suited for high-precision movements or controls, prevent users from performing highly precise operations.Aspects of the invention describe a novel input device that provides a significantly higher degree of precision and control for the user, reduces user fatigue, and is compatible with a range of tracking protocols commonly used in modern AR / VR systems. Some embodiments include a pen-like device designed to be held like a lightweight pen, featuring a balanced center of gravity, a design that keeps it aligned with the user's wrist during use, and a tracking area configured to maintain an unobstructed line of sight. This results in an ergonomically sound, high-precision input device that does not cause significant fatigue when used for extended periods.

[0028] In certain embodiments, the input device can be a stylus device designed for use in an augmented / virtual reality (AR / VR) environment, comprising a housing with a lower and an upper part, which, for example, is located in the Fig. Figure 3A-7 illustrates this. The housing may contain one or more processors and a communication module configured to establish wireless electronic communication between the pen device and at least one host computer device. In some cases, the lower part of the housing may be essentially straight and configured to be held by hand by a user during use of the input device. The lower part may include one or more input elements configured to generate control signals in response to activation by a user's hand, as well as a tip at one end of the lower part configured to act as an interface between the pen device and objects within the AR / VR environment.The upper part of the casing can be shaped as a hexagonal ring extending over three dimensions, with at least a portion of the ring having a hexagonal cross-section. The angle between the respective planes of the hexagonal ring and / or its cross-section can be approximately 30 degrees, without necessarily being symmetry (e.g., some angles may be greater or less than 30 degrees). When the pen is inserted lengthwise, the hexagonal ring can be curved downwards towards and directed towards a user's wrist, thus preventing the user's line of sight from being obstructed while operating the pen device. The hexagonal ring can have a series of planar facets, which may be oriented to point in different directions in three-dimensional space.Multiple emitters or sensors can be formed on some or all of the planar facets, thereby supporting tracking of the pen device in the three-dimensional space of the AR / VR environment. The use of emitters (e.g., infrared LEDs) and / or sensors (e.g., infrared detectors) may depend on the type of tracking infrastructure used by the AR / VR system. The planar facets can be oriented such that from any point 360 degrees axially to the orientation of the pen device (e.g., completely around the sides of the pen device), at least four of the planar facets are visible (e.g., detectable LED emission), although some embodiments may have facets arranged on other areas of the hexagonal ring such that at least three or four are visible from anywhere around the pen device. In some implementations, the center of gravity of the pen device may be located at the junction between the upper and lower parts.When using the pen-like device, the upper part is so compact that it does not extend laterally beyond the user's wrist during use (e.g., does not extend beyond the base of the arm). The lower part of the housing can be held by the user in a pen-grip configuration during use. To a person skilled in the art, many variations, modifications, and alternative embodiments beyond the non-limiting example described above are apparent. Definitions

[0029] A better understanding of the present revelation can be achieved by taking into account the following explanations.

[0030] The terms "computer simulation" and "virtual reality environment" used in this document can refer to virtual reality, augmented reality, mixed reality, or any other form of immersive, computer-simulated visual environment provided to a user. The terms "virtual reality" and "VR" used here can encompass a computer-simulated environment that recreates an imaginary scene. A user's physical presence in this environment can be simulated by allowing the user to interact with the scene and the objects depicted within it.Examples of VR environments include: a video game; a simulation program for a medical procedure, including a surgical or physiotherapy procedure; an interactive digital model of a design, including a computer-aided design; an educational simulation program, including an e-learning simulation; or other similar simulations. The simulated environment can be two- or three-dimensional.

[0031] The terms "augmented reality" or "AR," as used here, can encompass the use of computer-generated images displayed alongside a view of the real world. Examples of AR environments include: architectural applications that visualize real-world buildings; medical applications that provide users with additional information during surgery or therapy; and gaming environments that allow users to experience an enhanced simulation of the real world before entering a VR environment.

[0032] The terms "mixed reality" and "MR," as used here, can encompass the use of virtual objects presented as computer-generated images alongside a view of a real-world environment, where the virtual objects can interact with the real environment. The embodiments described below can be implemented in AR, VR, or MR environments.

[0033] The term "real environment" or "real world," as used here, can refer to the physical world (also referred to here as "physical environment"). Therefore, the term "real arrangement," when referring to an object (e.g., a body part or a user interface device), can refer to the object's arrangement in the real world and be referenced to a point. The term "arrangement," when referring to an object, can refer to a position (location and orientation). The position can be defined using a global or local coordinate system.

[0034] The term "computer-generated images" or "graphic images," as used here, can encompass images that are generated by a computer and displayed to a user as part of a virtual reality environment. The images can be displayed in two or three dimensions. The displays disclosed here can present images of a real-world environment, for example, by allowing a user to directly view the real-world environment and / or by presenting one or more images of a real-world environment (which might, for example, be captured with a camera).

[0035] The term "Head Mounted Display" or "HMD," as used here, can refer to a display of computer-generated images for a user. The HMD can comprise a graphical display positioned in front of a user, covering part or all of their field of vision. The display can be transparent, semi-transparent, or opaque. The HMD can be part of a headset. The graphical display of the HMD can be controlled by a display driver, which may include circuitry as defined here.

[0036] The term "electrical circuit" or "circuit," as used herein, may refer to, be part of, or include one or more of the following, or other suitable hardware or software components: a processor (shared, dedicated, or grouped); a memory (shared, dedicated, or grouped); a combinational logic circuit; a passive electrical component; or an interface. In certain embodiments, the circuit may include one or more virtual machines that can perform the described functionality. In certain embodiments, the circuit may include passive components, such as combinations of transistors, transformers, resistors, and capacitors, that can perform the described functionality.In certain embodiments, the circuit or functions associated with the circuit may be implemented using one or more software or firmware modules. In some embodiments, the circuit may include logic that can be implemented, at least partially, by hardware. The electrical circuit may be centralized or decentralized, including distribution across various devices that form part of the system or are in communication with the system, and which may include: a networked computer, including an external server; a cloud-based computer, including a server system; or a peripheral device.

[0037] The term “processor(s)” or “host / local processor(s)” or “processing resource(s)” as used herein may refer to one or more processing units, including an application-specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device (PLD), a microcontroller, a field-programmable gate array (FPGA), a microprocessor, a digital signal processor (DSP), or any other suitable component. A processor may be configured to use machine-readable instructions stored in memory.The processor can be centralized or decentralized, including distribution across various devices that form part of the system or communicate with the system. These devices can include: a networked computer, including an external server; a cloud-based computer, including a server system; or a peripheral device. The processor can be located in a peripheral device (e.g., a pen device), which may include a user interface device and / or a head-mounted display (HMD), and / or in a computer (e.g., a personal computer or similar device), and / or in another device that communicates with a computer system.

[0038] The term "computer-readable medium(s)," as used here, can include conventional non-volatile storage devices, such as random-access memory (RAM), optical media, hard disks, flash drives, memory cards, floppy disks, optical drives, and / or combinations thereof. It is assumed that, while one or more storage devices may be located in the same physical location as the system, they may also be located outside the host system and communicate with the processor(s) via a computer network. Furthermore, if more than one storage device is used, the first may be located in the same physical location as the host system, while other storage devices may be located outside the host system. The physical location(s) of the storage device(s) may change.In addition, one or more storage systems can be implemented as "cloud storage" (i.e., one or more storage systems can be partially or fully network-based or accessible via the network).

[0039] The term "communication resources," as used here, can refer to hardware and / or firmware for electronic information transmission. Wireless communication resources can include hardware for sending and receiving signals via radio waves and can incorporate various protocol implementations, such as the 802.11 standards described in the Institute of Electronics Engineers (IEEE), Bluetooth™, ZigBee, Z-Wave, Infrared (IR), RF, or similar protocols. Wired communication resources can include a modulated signal carried over a signal line, with the modulation being consistent with a serial protocol such as Universal Serial Bus (USB), Serial Peripheral Interface (SPI), Inter-Integrated Circuit (I2C), RS-232, RS-485, or other protocol implementations.

[0040] The term “network” or “computer network”, as used herein, may include one or more networks of any type, including a public land mobile network (PLMN), a telephone network (e.g., a public switched telephone network (PSTN) and / or a wireless network), a local area network (LAN), a wide area network (MAN), a wide area network (WAN), an Internet Protocol multimedia subsystem (IMS) network, a private network, the Internet, an intranet and / or any other suitable network type.

[0041] The term "sensor system," as used here, can refer to a system that possesses functionality for capturing positional information relating to input devices, peripherals, and other objects in a physical world, which may include a body part or another object. The term "tracking system" can refer to the detection of movements of these objects. A body part can include an arm, a leg, a torso, or a subset thereof, including a hand or finger (thumb or other fingers). A body part can include a user's head. The sensor system can provide positional information from which a user's gaze direction and / or field of view can be determined. The object can include a peripheral device that interacts with the system. The sensor system can provide a real-time stream of positional information.One embodiment provides an image stream that can represent a user's avatar. The sensor and / or tracking system may include one or more of the following components: a camera system; a magnetic field-based system; capacitive sensors; radar; acoustics; other suitable sensor configurations; optical, radio, magnetic, and inertial technologies, such as lighthouses, ultrasound, IR / LEDs, SLAM tracking, light detection and ranging (LIDAR) tracking, ultra-wideband tracking, and other suitable technologies known to a person skilled in the art. The sensor system may be arranged on one or more of the following devices: a peripheral device, which may include a user interface device, the HMD; a computer (e.g., a PC, a system controller, or a similar device); or another device that communicates with the system.

[0042] The term "camera system," as used here, can refer to a system comprising a single instance or multiple cameras. The camera can be a 2D camera and / or a 3D camera and / or an infrared (IR) camera and / or a time-of-flight (ToF) camera. The camera can have a complementary metal-oxide semiconductor (CMOS) sensor, a charge-coupled device (CCD) image sensor, or any other type of optical sensor commonly used for image generation. The camera can have an IR filter that can be used for object tracking. The camera can include a red-green-blue (RGB) camera that can be used to generate real-world images for augmented reality (AR) or mixed-reality (RRP) simulations. In one embodiment, instead of separate cameras, different images from a single camera can be processed alternately, for example, with an IR filter and for RGB.Images from more than one camera can be combined to obtain a field of view corresponding to that of a user. A camera system can be located on any component of the system. In one embodiment, the camera system is located on a headset or HMD, with one recording area of ​​the camera system capable of capturing a user's field of view. Additional cameras can be positioned elsewhere to track other parts of a user's body. The use of an additional camera or cameras to cover areas outside the user's immediate field of view can be advantageous because it allows for pre-computer generation (or the early initiation of other computations) that can be used to computer-generate these areas.The camera system links the user's body parts within the augmented or virtual reality environment, thereby enhancing the performance perceived by the user in the virtual reality simulation (e.g., a more immediate reaction). The camera system can provide an application program with information, potentially including an image stream, from which the application program can deduce the position and orientation. The application program can implement known object tracking techniques, such as feature extraction and identification.

[0043] The term "user interface device," as used here, can encompass various devices that form an interface between a user and a computer. Examples of such devices include: pointing devices, including those based on the movement of a physical object, such as a mouse, trackball, joystick, keyboard, gamepad, steering wheel, paddle, yoke (control column for an aircraft), D-pad, engine control console, pedals, light gun, or button; pointing devices based on touching or being near a surface, such as a stylus, touchpad, or touchscreen; or a 3D motion controller. The user interface device may have one or more input elements.In certain embodiments, the user interface device may include devices that the user carries. "Carries" may refer to the user holding the user interface device in a manner other than by grasping it with their hands. In many of the embodiments described here, the user interface device is a pen-like device designed for use in an AR / VR environment.

[0044] The term "IMU," as used here, can refer to an inertial measurement unit capable of measuring motion in six degrees of freedom (6 DOF) along the Cartesian x, y, and z coordinates and about three axes—pitch, roll, and yaw. In some cases, certain implementations may use an IMU that detects motion with less than 6 DOF (e.g., 3 DOF, as explained in more detail below).

[0045] The term "keyboard," as used here, can refer to an alphanumeric keyboard, an emoji keyboard, a graphical menu, or any other collection of characters, symbols, or graphical elements. A keyboard can be a physical, mechanical keyboard or a touchpad keyboard, such as the on-screen keyboard (OSC) of a smartphone or tablet. Alternatively, it can also refer to a virtual keyboard presented in an AR / MR / VR environment.

[0046] The term "fusion," as used here, can refer to a combination of different positioning techniques and / or positioning techniques based on different coordinate systems, for example, to enable a more precise determination of an object's position. For instance, data from an IMU and a camera tracking system, both tracking the movement of the same object, can be fused. A fusion module, as described here, performs the fusion function using a fusion algorithm. The fusion module can also perform other functions, such as combining position or motion vectors from two different coordinate systems or measurement points into a single vector.

[0047] It should be noted that various embodiments of the input devices described herein often refer to a "lower part" and "upper part," as described in more detail below. It should be noted that the lower part (the part usually held by a user) can also be referred to as the "first part," and the two terms are interchangeable. Likewise, the upper part (the part that usually contains the sensors and / or emitters) can be interchangeably referred to as the "second part." Typical AR / VR system environment

[0048] The basic elements of a typical AR / VR system include a device that allows a user to immerse themselves in an AR / VR environment (e.g., a head-mounted display (HMD), a holographic projector, an audio system, haptic feedback, or similar devices), one or more input devices (e.g., a stylus, a remote control, a joystick, a pistol-grip controller, etc.) through which the user can connect to the AR / VR environment, and a tracking system to monitor the user's position, which may include tracking the location of the HMD, the input device, and / or other devices (e.g., wearables, etc.). One or more computing devices (e.g., desktop computers, laptop computers, gaming platforms, entertainment / media systems, cloud-based systems, combinations thereof, etc.) may be used to communicate with and control the various input and output elements.

[0049] Several different types of tracking systems exist that can be used to track the position and orientation of an input device, depending on the specific design. "Outside-in" systems typically use external devices to track the headset (e.g., head-mounted display) and accessories (e.g., stylus). These external tracking devices (e.g., cameras, IR sensors, etc.) can be distributed throughout the space and are generally directed at the HMD and / or the input device. They are designed to determine the position and orientation of the HMD and input device relative to the AR / VR environment. Outside-in systems can exhibit very low latency and high accuracy, which can be further improved by integrating additional tracking devices.Some of the disadvantages associated with outside-in systems are that they must remain within a defined field of view, and the problem of obstruction, since outside-in tracking is typically based on line-of-sight tracking. Therefore, outside-in systems typically need to be suitable for 360-degree tracking; otherwise, there is a risk that the tracking position or orientation of the HMD or input device will be lost if the user turns or moves in certain positions or orientations.

[0050] In "inside-out" tracking systems, a camera or sensor array is mounted on the device being tracked (e.g., HMD, stylus device), and changes in its position and / or orientation relative to the AR / VR environment are detected. When the tracked device is moved, the new tracking coordinates are updated in real time. In some cases, inside-out tracking can be supplemented with reference markers to improve position / orientation tracking. One advantage is that a user has greater freedom of movement within the virtual world without the need for sensors distributed throughout the space. One disadvantage is that all calculations are performed within the tracked system itself, for example, by the HMD, which can lead to performance limitations.To a person skilled in the art, many variations, modifications, and alternative embodiments of the various available AR / VR systems are apparent from this disclosure, as is how embodiments of the input devices described herein (e.g., pen devices) can function in these different systems. One of these embodiments is described below with reference to... Fig. 1 shown and described.

[0051] Fig. Figure 1 shows a user 102 operating a pen device 110 according to certain embodiments in an "outside-in" AR / VR environment 100. The AR / VR environment 100 can include a computer 140 and any number of peripheral devices, such as a display device 142, a computer mouse 144, a keyboard 146, or another input / output device. The depicted user 102 wears a head-mounted display 160 and uses a pen 110 to draft a letter on a virtual parchment 165. The pen can be in wireless electronic communication with one or more external sensors 130 (1, 2, ...n), the HMD 160, the computer 140, or any combination thereof. Similarly, the HMD 160 can be in wireless electronic communication with one or more external sensors 130 (1, 2, ...n), the computer 140, the pen 110 or any combination thereof.

[0052] As in Fig. As shown in Figure 1, the user can use the pen 110 with high precision due to its exceptional ergonomic properties, which allow for longer use with less fatigue, as shown below with reference to the Fig. 6A-6C is described in more detail. An outside-in system, such as the one shown, can have emitters (e.g., IR LEDs) arranged on pin 110, as described below with reference to Fig. 7 is described in more detail. Alternatively or additionally, the pen 110 can have multiple sensors for inside-out tracking that can perform motion and orientation calculations locally (e.g., using the processor(s) 210) or externally (e.g., using the HMD 160, computer 140, etc.). A person skilled in the art, with knowledge of this disclosure, can see many variations, modifications, and alternative embodiments of a tracking pen 110 for various common AR / VR tracking systems. Simplified embodiment of a system for an AR / VR input device

[0053] Fig. Figure 2 shows a simplified system block diagram (“System”) 200 of an input device 110 according to certain embodiments. The System 200 can comprise one or more processor(s) 210, an input acquisition block 220, a motion tracking block 230, a power management block 240, and a communication block 250. Each of the System blocks 220-250 can be electrically connected to the Processor 210. The System 200 can also comprise other systems, which are not shown or explained here for the sake of clarity in illustrating the new features described herein.

[0054] In certain embodiments, the processor(s) 210 may comprise one or more microprocessors (µCs) and be configured to control the operation of the system 200. Alternatively or additionally, the processor 210 may comprise one or more microcontrollers (MCUs), digital signal processors (DSPs), or the like, with supporting hardware, firmware (e.g., memory, programmable I / Os, etc.), and / or software, as is naturally known to a person skilled in the art. Alternatively, the MCUs, µCs, DSPs, ASICs, programmable logic devices, and the like may be implemented in other system blocks of the system 200. For example, the communication block 250 may include a local processor for controlling communication with the computer 140 (e.g., via Bluetooth, Bluetooth LE, RF, IR, wired, ZigBee, Z-Wave, Logitech Unifying, or another communication protocol).In some embodiments, the performance of System 200 (e.g., speed and bandwidth) can be increased by means of multiple processors; however, multiple processors are not required and are not necessarily relevant to the novelty of the embodiments described herein. Alternatively or additionally, as is of course known to a person skilled in the art, certain aspects of the processing can be carried out using an analogous electronic design.

[0055] The input capture block 220 can capture a key press (e.g., of the operating elements 416 of Fig. 4A, buttons, triggers, etc.), a scroll wheel and / or trackball control (e.g., rotation detection), sliders, switches, tactile sensors (e.g., one- and / or two-dimensional touchpads), force sensors, and the like. The activated input element (e.g., a pressed button) can generate a corresponding control signal (e.g., a human-device interface (HID) signal) to control a computer device (e.g., computer 160) that is communicatively connected to the input device 110 (e.g., instantiating a "grab" function in the AR / VR environment). Alternatively, the functions of the input acquisition block 220 can be combined by the processor 210 or in combination with it.

[0056] In some embodiments, the input sensing block 220 can detect a touch or a touch gesture on one or more touch-sensitive surfaces as described above. The input sensing block 220 can have one or more touch-sensitive surfaces or touch sensors. Touch sensors generally have sensor elements suitable for detecting a signal such as direct contact, an electromagnetic or electrostatic field, or a beam of electromagnetic radiation. Touch sensors can typically detect changes in a received signal, the presence of a signal, or the absence of a signal. A touch sensor can have a source for emitting the detected signal, or the signal can be generated by a secondary source.Touch sensors can be configured to detect the presence of an object at a distance from a reference zone or point (e.g., < 5 mm), in contact with a reference zone or point, or a combination thereof. Certain embodiments of the input device 110 may or may not use touch detection or touch sensor elements.

[0057] According to some aspects, the input detection block 220 can control the function of haptic devices implemented on an input device. For example, the input detection block 220 can receive and process input signals generated by haptic devices. An input signal could be, for example, an input voltage, charge, or current generated by a piezoelectric device in response to a force applied to its surface (e.g., touch by the user). In some embodiments, the input detection block 220 can control an output from one or more haptic devices of the input device 14. Thus, for example, the input detection block 220 can control certain parameters that define the characteristics of the haptic feedback.Some input and output parameters can include a pressure threshold, a release threshold, feedback intensity, feedback force amplitude, feedback duration, feedback frequency, overvoltage (e.g., when using different voltage levels at different stages), and temporal feedback modulation. Alternatively, haptic input / output control can be performed by the Processor 210 or in combination with it.

[0058] The input acquisition block 220 can incorporate touch and / or proximity sensor functionality. Examples of touch / proximity sensor types include, but are not limited to, resistive sensors (e.g., based on standard 4-wire air gap sensors, based on carbon-loaded plastics that exhibit different electrical properties depending on pressure (FSR), interpolated FSR, etc.), capacitive sensors (e.g., surface capacitance, intrinsic capacitance, mutual capacitance, etc.), optical sensors (e.g., infrared light barrier matrix, laser-based diode coupled with photodetectors to measure the travel time of light, etc.), acoustic sensors (e.g., piezoelectric buzzers coupled with microphones to detect changes in wave propagation patterns relative to touch points, etc.), and the like.

[0059] The motion tracking block 230 can be configured to track or enable tracking of a movement of the input device 110 in three dimensions within an AR / VR environment. For outside-in tracking systems, the motion tracking block 230 can incorporate multiple emitters (e.g., IR LEDs) as described, for example, in Fig. 7 shown, which are arranged on the input device, have reference markers or other tracking aids so that the outside-in system can track the position, orientation, and movement of the input device within the AR / VR environment. For inside-out tracking systems, the motion tracking block 230 can have multiple cameras, IR sensors, or other tracking aids so that the inside-out system can track the position, orientation, and movement of the input device within the AR / VR environment. To ensure accurate tracking, the tracking aids (also referred to as "tracking elements") are preferably designed in both cases such that at least four reference points can be determined on the input device at any given time.Some embodiments may include emitters and sensors, reference markers, or another combination of several tracking aids, so that the input device can be used in both inside-out and outside-in tracking systems without further modification ("out of the box"). Such embodiments can be used more universally and independently across multiple system platforms.

[0060] In certain embodiments, an inertial measurement unit (IMU) can be used to supplement motion detection. IMUs can consist of one or more accelerometers, gyroscopes, or the like. Electromechanical devices (e.g., microelectromechanical systems (MEMS)) designed to measure acceleration forces (e.g., static and dynamic forces) can serve as accelerometers. One or more accelerometers can be used to detect three-dimensional (3D) positioning. For example, a three-axis accelerometer or two two-axis accelerometers can be used in 3D tracking. Accelerometers can also determine the velocity, spatial orientation, and acceleration of the input device 110 in 3D space.In some embodiments, gyroscopes can be used instead of or in conjunction with the accelerometer(s) to determine the movement or orientation of the input device in 3D space (e.g., as applied in a VR / AR environment). As can be seen by an average person skilled in the art, any suitable type of IMU and any number of IMUs can be integrated into an input device 110.

[0061] The power management block 240 can be configured to handle power distribution, charging, energy efficiency, and the like at the input device 110. In some embodiments, the power management block 240 can include a battery (not shown), a USB-based charging system for the battery (not shown), and a power supply network within the system 200 to power each subsystem (e.g., the communication block 250, etc.). In certain embodiments, the functions of the power management block 240 can be integrated into the processor(s) 210. Some alternative embodiments may not have a dedicated power management block. For example, functional aspects of the power management block 240 may be combined with or integrated into another block (e.g., processor(s) 210).

[0062] The communication block 250 can be configured, according to certain embodiments, to enable communication between the input device 110 and the HMD 160, the computer 140, or other devices and / or peripherals. The communication block 250 can be configured to enable a wireless connection using any suitable communication protocol (e.g., radio frequency (RF), Bluetooth, BLE, infrared (IR), ZigBee, Z-Wave, Logitech Unifying, or a combination thereof).

[0063] Although certain systems are not explicitly described, they are, as would be apparent to a person skilled in the art, to be considered part of System 200. For example, System 200 may include a bus system for transferring power and / or data to and from the various systems. In some embodiments, System 200 may include a memory subsystem (not shown). A memory subsystem may store one or more software programs to be executed by processors (e.g., in processor(s) 210). It should be noted that “software” may refer to sequences of instructions which, when executed by processing unit(s) (e.g., processors, processing devices, etc.), cause System 200 to perform certain software program functions.Instructions can be stored in the form of firmware located in read-only memory (ROM), and / or in the form of application programs stored in media memory, which can be read into memory for execution by processing devices. Software can be implemented as a single program or as a collection of individual programs and can be stored in non-volatile memory and copied, in whole or in part, into volatile memory during program execution. Processing devices can retrieve program instructions from a memory subsystem, which must be executed to perform various functions (e.g., software-controlled automatic source setting, etc.) as described herein.

[0064] It should be noted that System 200 serves for illustrative purposes and that, as is apparent to a person skilled in the art, many variations and modifications are possible. System 200 may include further functions or features not explicitly described here (e.g., mobile phone, Global Positioning System (GPS), power management, one or more cameras, various ports for connecting external devices or accessories, etc.). Even when System 200 is described with reference to specific blocks (e.g., input acquisition block 220), this should be understood as meaning that these blocks are defined to explain specific embodiments of the invention and not to restrict embodiments to a particular physical configuration of components. The individual blocks need not correspond to physical components.Blocks can be configured to perform various functions, such as programming a processor or providing suitable processes, with some blocks being reconfigurable or not, depending on their initial configuration. Certain embodiments can be implemented in various devices, including electronic devices realized using any combination of circuitry and software. Furthermore, aspects and / or parts of the system 200 can be combined with or operated by other subsystems, depending on the design. For example, the power management block 240 and / or the motion tracking block 230 can be integrated into the processor(s) 210 instead of functioning as a separate unit. Certain embodiments of the input device

[0065] As described above, some input devices for modern AR / VR systems feature donut- or ring-shaped tracking geometries housed in a separate pistol grip. The pistol grip offers ergonomic features that allow users to comfortably handle larger geometric shapes and greater weight than would be comfortable for a user to hold and control with a pen grip (such as a pinch grip) between their thumb and forefinger.Circular or donut shapes can provide a good geometry for an array of sensors covering a 360-degree field of view. These are typically positioned above or around the hand / grip to ensure that the sensors are not blocked or obscured by the hands, making the pistol grip a good choice. However, this comes at the expense of the nimble dexterity and precise control described above. Certain embodiments of the invention include input devices that have a pen-like shape with reduced size, weight, and surface area, while offering better balance and excellent tracking characteristics. These devices can be used continuously like a pen for extended periods with minimal fatigue (e.g., 4 hours or more) and can be held like a pen to allow for better control and more precise movement.Some non-restrictive implementations are in the . Fig. 3A-3C shown.

[0066] Fig. Figure 3A shows a pen device (“pen”) 310(1) designed for operation in an AR / VR environment, according to certain embodiments. The pen device 310(1) has a lower part (also referred to as the “front part”) 312(1), by which it is typically held by a user when using the pen 310(1), and an upper part (also referred to as the “rear” or “back part”) 314(1) with a hexagonal ring geometry designed for excellent tracking properties, as shown below with reference to the Fig. 4A-7 will be explained in more detail. Fig. Figure 3B shows a pin 310(2) with a similar lower part 312(2) and an upper part 314(2) which has a narrower hexagonal ring geometry. Fig. 3C-3E disclose a pen 310(3) with a flatter profile of the lower part 312(3), wherein the upper part 314(3) is shaped like a question mark. Each embodiment has several planar facets oriented in different directions, which may have sensors (e.g., IR sensors) and / or emitters (e.g., IR LEDs) arranged thereon to assist in tracking the position, orientation, and movement of the pen device in the AR / VR environment, as described below with reference to Fig. 7 is described in more detail.

[0067] Alternatively or additionally, the ones in the Fig. The pen devices shown in Figures 3A-3C or described in this document can be held in alternative positions. For example, a user can hold the pen in a pistol-grip position with the lower part 312(1) pointing downwards and the upper part 314(1) pointing upwards to create an adaptable input device that can be used both as a pen and as an enhanced pistol-grip interface. Many variations, modifications, and alternative embodiments thereof are apparent to a person skilled in the art with knowledge of this disclosure.

[0068] Fig. Figure 4A shows aspects of a pen device (“pen”) 410 for use in an AR / VR environment, according to certain embodiments. The pen 410 may have a lower part 412 with one or more input elements 416. The lower part 412 may be pen-shaped, allowing the user to, for example, use it in a pincer grip (see, e.g., Figure 4A). Fig. 5A) can hold and handle to improve control, dexterity, and precision compared to pistol-grip controllers. Typically, the lower part of the 412 is shaped to be essentially straight, as in Fig. The lower part 412 is shown in Figure 4A, but it can also have non-even sections, curves, or cross-sections. The input elements 416 can include any number of buttons, controls, touch-sensitive sensors, or other input elements configured to generate control signals in response to actuation by a user's hand. The lower part 412 can have a tip configured for handling as an interface between the pen device and objects within the AR / VR environment, which may include a pen-like function, a pointer, a "spray nozzle" in photo-editing software, or another application. In some embodiments, the tip can include an active sensor (e.g., a pressure sensor) so that the pen 410 can detect contact or impact with real-world objects and / or surfaces.Some or all of the input elements 416 can be controlled by the input acquisition block 220, the processor 210, or a combination of these, as described above with reference to . Fig. 2 was described.

[0069] In some embodiments, the upper part 414 of the housing can have any shape, such as the shape of a ring (see e.g. Fig. 3A, 3B, 4A-C) or a projection (see e.g. Fig. 3C), which is preferably not straight and is curved to extend in three dimensions to ensure accurate tracking characteristics, as described below. In some embodiments, the upper part 414 may comprise a polygonal ring (e.g., a hexagonal ring, a rhombic ring, an octagonal ring, a pentagonal ring, a hemispherical ring, a circular ring, etc.) extending in three dimensions and may have a polygonal cross-section (e.g., a hexagonal cross-section, a triangular cross-section, etc.) through at least a portion of the polygonal ring. It should be noted that the figures show a hexagonal ring and a hexagonal cross-section, and are referred to as such for illustrative purposes. However, as will be apparent to a person skilled in the art, the concepts described herein can also apply to other polygonal shapes and cross-sections.In some embodiments, the angle between the respective planes (e.g., between each curvature) of the hexagonal ring and / or its cross-section may be approximately 30 degrees, without needing to be symmetrical (e.g., some angles may be greater or less than 30 degrees). Other angles are possible (e.g., 45 degrees, 60 degrees, 15 degrees, etc.), and various angles may be used throughout the upper part 414 or any part of the pen 410. The hexagonal ring may have a section that is bent longitudinally downward in a line collinear with the first part of the housing (e.g., in line with a user's wrist when the pen is in use), thereby preventing the user's line of sight from being obstructed when operating the pen device, as described below. In general, sharper bends (e.g.,Bends of 30 degrees or more cause sensors / emitters to be positioned further apart in three-dimensional space and thus easier to distinguish, enabling simpler and more reliable tracking (e.g., differentiating between four or more detected sensors / emitters). Conversely, shallow bends (e.g., less than 20 degrees) cause sensors / emitters to be positioned closer together in three-dimensional space and may be more difficult to track, resulting in less reliable tracking. A bend of 30–60 degrees is typically preferred, although bends as low as 15 degrees can function satisfactorily in robust detection systems. In some cases, while the use of very steep bends can improve tracking characteristics, it can also lead to ergonomic issues such as obscuration (see, e.g., [reference]). Fig. 6A), unfavorable weight distribution (see e.g. Fig. 6B-6C) or similar.

[0070] The polygonal ring can have multiple planar facets, which can be oriented to point in different directions in three-dimensional space. Multiple emitters or sensors can be positioned on some or all of the planar facets, facilitating tracking of the pen device in three-dimensional space within the AR / VR environment. The use of emitters (e.g., infrared LEDs) and / or sensors (e.g., infrared detectors) may depend on the type of tracking infrastructure used by the AR / VR system. The planar facets can be oriented such that from any point 360 degrees axially to the orientation of the pen device (e.g., completely around the sides of the pen device), at least four of the planar facets are visible.detectable LED emission), with some embodiments also having facets arranged on other areas of the hexagonal ring such that at least 3 or 4 are visible from somewhere around the pen device. As stated above, the ring extending in three dimensions ensures that emitters and / or sensors can be "seen" (i.e., detected via line of sight) from any point 360 degrees around the device. It should be noted that a user may obstruct the line of sight between the input device's sensors / emitters and external tracking elements during use; therefore, ensuring that many points (e.g.,If four or more sensors / emitters are detectable at the input device, the probability of reliable, continuous, and uninterrupted tracking is improved, as the unavoidable obstruction of some sensors / emitters by the user (e.g., body, arms, etc.) when using the input device can be compensated for by additional sensors / emitters that can still be detected and tracked simultaneously. Fig. 4B the hexagonal ring extends along the x-axis 450 and the y-axis 452 of a Cartesian coordinate space. Fig. Figure 4C shows how the pin 410 extends along the z-axis 454. In some embodiments, the facets along the ring can be symmetrical or asymmetrical. In some cases, the asymmetrical arrangement of facets may be preferable for tracking purposes, since each combination of detected sensors / emitters (which may be referred to as "tracking points" for clarification) represents a unique constellation, whereas with a symmetrical arrangement, multiple possibilities exist.

[0071] Obviously, the ones in the Fig. The facets shown in Figures 3A-4C are not necessarily formed on every section of the upper part 414. Some embodiments may have facets that are formed completely around the upper part 414; more or fewer facets; facets with different polygonal shapes, sizes, and dimensions; continuous or discontinuous patterns of facets; or any combination thereof, as would be apparent to a person skilled in the art with knowledge of this disclosure.

[0072] In some cases, embodiments may have smooth and / or curved surfaces instead of facets. In such cases, some embodiments may have inlaid (or non-inlaid) window surfaces for mounting sensors and / or emitters. To a person skilled in the art, many variations, modifications, and alternative embodiments thereof are apparent from this disclosure.

[0073] In some implementations, the sensors / emitters of the input device 410 can be configured to be used universally in inside-out systems, outside-in systems, room-scale systems, world-scale systems, systems that detect / see three or more sensors / emitters on the input device 410 from a position laterally to the input device 410 (e.g., "Lighthouse"-based systems), or systems that can detect three or more sensors / emitters from behind (e.g., HMD-based detection), or the like. Some embodiments may be optimized for one, more than one, or all of these systems, including systems not expressly mentioned herein but which, within the scope of this disclosure, may be considered as known to a person skilled in the art.

[0074] In some implementations, the center of gravity of the pen device may be located at the transition between the upper and lower parts. When in use, the upper part is so compact that it does not extend laterally beyond the user's wrist (e.g., does not reach beyond the base of the arm). To a person skilled in the art, many variations, modifications, and alternative embodiments of this disclosure are apparent.

[0075] In certain embodiments, part or all of the system 200 can be integrated into the pen 410 or one of the novel input devices described herein (e.g., pen 110, 310, 410, 510, 610, 710). For example, the pen 410 can incorporate (a) processor(s) 210 and the communication block (“module”) to establish a wireless electronic communication link between the pen device and at least one host computer device (140).

[0076] In further embodiments, the input device 410 can be used in environments other than VR / AR / MR. For example, the position, orientation, movement, and operation of the input device 410 can also be tracked for use with a smart whiteboard in a classroom scenario or other real-world application where tracking can be used in a non-VR / AR / MR environment. In some embodiments, an input device can comprise a tracking section (e.g., a polygonal ring with sensors / emitters configured as described above) without a lower part (e.g., without a pen grip or other user-handleable section). Many variations, modifications, and alternative embodiments thereof are apparent to a person skilled in the art with knowledge of this disclosure. Versatile uses for the handle of the input device

[0077] The input devices described here can offer excellent control, dexterity, and precision for a wide variety of applications. Furthermore, the input devices can be held in various ways to suit the user's preferences.

[0078] Fig. Figure 5A shows a pincer grip for connection with the pen device 510, according to certain embodiments. It is shown how the user's hand 504 holds the lower part 512 of the pen 510 between the thumb and fingers (e.g., index and middle fingers), while the upper part 514 rests on a part of the user's hand between the thumb and index finger (a "purlicue"; saddle between thumb and index finger). In a preferred grip position, a user may use only the index finger or three or more fingers. The user can grip the lower part 512 further up or down as desired.

[0079] Fig. Figure 5B shows a second grip position for connecting to the input device 510, according to certain embodiments. It shows how the user's hand 504 holds the lower part 512 of the stylus 510 in a different way, in which the stylus 510 is supported rather than gripped, so that the index finger and thumb align the stylus while the stylus is supported (rests) on the saddle between the thumb and index finger of the hand 504. This grip may be useful for applications requiring pointing or aiming, such as a virtual projectile in a game application, a virtual telescope or binoculars, or other suitable applications. To a person skilled in the art, many variations for connecting to the stylus 510 are apparent with knowledge of this disclosure. Ergonomic advantages of certain embodiments of the input device

[0080] When used in an AR / VR environment, the pen offers a number of ergonomic advantages over other controller types (e.g. remote controls, pistol grip controllers, etc.).

[0081] Fig. Figure 6A shows aspects of certain ergonomic advantages of the pen device 610, according to certain embodiments. It depicts a user 602 drawing a landscape with the pen 610 on a projected virtual canvas 640 (displayed by HMD 660). It is noted that the pen 610 is held at eye level, yet the user's line of sight 609 is not obstructed. With conventional devices, such as pistol-grip controllers, where tracking mechanisms ("donuts") are formed around or above the hand, an unobstructed line of sight at eye level may not be possible because the tracking devices can interfere with the user's view. An unobstructed line of sight 609 is made possible, in part, by the way the tracking section (e.g., upper part 514) curves away from the line of sight while maintaining its three-dimensional extent.Although the embodiments presented here show that the tracking section of various pen devices curves first upwards and then downwards, other embodiments are of course possible that still extend in three dimensions. For example, some embodiments may first curve downwards and then upwards, some may not have a complete ring or may be more circular, or may assume other polygonal shapes (e.g., square, pentagram, octagon, etc.). Many variations, modifications, and alternative embodiments of this are apparent to a person skilled in the art with knowledge of this disclosure.

[0082] Fig. Figure 6B shows further aspects of certain ergonomic advantages of the pen device, according to certain embodiments. Fig. Figure 6B shows that the pin 610 is advantageously balanced with its center of gravity 616 at the transition between the upper part 612 and the lower part 614, such that the pin 610 can be cantilevered in the space between the index finger and thumb of a user's hand (purlicue) 606 when the user's hand is positioned in an operating position as shown. It is noted that the center of gravity can be shifted forward or backward to accommodate the weight, length, or thickness of the housing, the distribution of the various electronic components within the pin 610 (e.g., System 200), the length, configuration, dimensions, and orientation of the projections (e.g., hexagonal ring) of the upper part 614, or other features. Some embodiments may have features other than those shown in Figure 6B. Fig. Figure 6B illustrates this, and a similar center of gravity 616 can be achieved by adjusting the various parameters listed above. One advantage of a balanced center of gravity is better control of the pen 610, as the user does not have to contend with gravity pulling the pen forward or backward, which can be challenging when attempting precise movements. In contrast, modern pistol-grip controllers are often heavy, unwieldy, and have significant problems compensating for weight, especially when handled upside down in a manner similar to that used with the pen 610.For example, the weight, large surfaces, and large geometric parts would distribute the weight unfavorably, overload the user's hand stabilizing muscles as they compensate for forward and backward or left and right tilting, and make precise control of the tip difficult, leading to poor precision and rapid fatigue. A balanced center of gravity can reduce hand fatigue, allowing the user to use the pen continuously for extended periods. In some embodiments, the 610 pen can be balanced longitudinally and laterally (i.e., lengthwise and crosswise).

[0083] Fig. Figure 6C shows further aspects of the ergonomic advantages of the pen device, according to certain embodiments. Fig. Figure 6C shows a top view of the pen device during operation by a user. The user's hand 604 holds the pen 610 such that the pen 610 is aligned with the user's wrist 608. It is noted that when using the pen device, the upper part of the pen 610 does not extend laterally beyond the user's wrist. For example, the hexagonal ring does not extend beyond the wrist. In some embodiments, small portions of the hexagonal ring may extend slightly beyond the user's wrist 608 (e.g., 1-2 cm), but the majority of the ring's weight remains centered over the user's wrist.This can be ergonomically advantageous, as the user does not have to struggle with gravity-induced forces pulling downwards on both sides of the pin device, which can cause unintentional tipping or lead to premature fatigue during prolonged use. Tracking geometries for specific designs

[0084] To ensure smooth, six-degree-of-freedom movement of input devices in virtual and augmented reality systems, these devices should be tracked in three-dimensional space according to a coordinate system corresponding to the sensors / emitters and / or report their position. To guarantee accuracy, the sensors should ideally report unambiguous, non-planar four-point references to the system software continuously and across all rotation angles that typically occur due to the user's various movements. The range of motion can result from performing activities such as, but not limited to, drawing, painting, writing, pointing, etc., in free space while holding the device, for example, with a thumb in opposition in a pincer grip. The system should be able to detect the device's position, direction, and orientation.To prevent the sensors from being obscured, for example by the hand holding the shape relative to "room-scale system lighthouses", the sensor housing can be geometrically positioned at the back of the device, where it extends beyond the user's wrist (see e.g. ). Fig. 6A-C). This geometric shape and its relative behavior within the system are further subject to the desire to minimize ergonomic factors such as size and weight, and to the advantage of a balanced design. Some embodiments of the described shapes have been shown in earlier figures presented here.

[0085] When the input device is static (not moving), the AR / VR tracking system uses three unique points to determine the device's orientation and position. If the system loses sight of any one of these three points, further tracking may fail. To ensure continuous tracking during movement / rotation, the system should be able to continuously capture four points to allow the transition from one unique set of three points to another. To meet this performance parameter, some embodiments incorporate a geometric shape (e.g., a hexagonal ring with a hexagonal cross-section) that forms a closed three-dimensional ring, as described in the Fig. Figures 6B-6C show a strong bend in each of the X, Y, and Z axes of a Cartesian coordinate system. In some cases, these geometries result in a series of distinct, non-planar sensor positions with overlapping viewing cones that cover 360 degrees and extend from the input device. In other words, the system should be able to detect the input device in any position and orientation. In some embodiments, the shape may be symmetrical to satisfy ergonomic factors (e.g., balance), while the sensor positions may be asymmetrical to aid pattern recognition. Each sensor may also have a distinct angle to further differentiate between sensors or emitters formed on the left and right sides. Some implementations feature facets that allow for distinct angles and planar "windows."Another advantage of predefined facets is that sensors and / or emitters with normal or even below-average manufacturing tolerances can be mounted and are still correctly aligned on the comparatively large mounting surface.

[0086] Fig. Figure 7 shows aspects of the configuration and functional properties of the sensor / emitter array of a pen device 710, according to certain embodiments as described above. The pen 710 has a lower part 712 and an upper part 714, which may be similar to the embodiments shown and described above, thus simplifying the explanation of the following concepts of the invention. The upper part 714 may have a hexagonal ring equipped with several polygonal planar facets 718. Some or all of the planar facets 718 may be equipped with a sensor (e.g., an IR sensor), an emitter (e.g., an IR LED), a reference marker(s), and / or another tracking element that can be used to support the tracking of the pen 710 in the three-dimensional space of an AR / VR environment.In some embodiments, the multiple planar facets can be oriented such that, from any point 360 degrees axially to the orientation of the pen device, at least four of the sensors / emitters formed on their respective planar facets are visible. As the pen 710 moves and rotates, the at least four sensors / emitters can change. For example, from an external sensor, an initial set of four or more sensors / emitters may be visible, and as the pen rotates, part of the initial set of four or more sensors / emitters may still be visible while others are no longer visible and new ones become visible, so that the tracking criterion of at least four or more sensors / emitters remains satisfied to ensure continuous and accurate tracking. In some cases, the emitters can transmit their respective signals over any suitable area (e.g., 15-degree cone, 30-degree cone, etc.).) radiate in such a way that signal overlap and wider coverage can occur, thus requiring fewer emitters to achieve 360-degree tracking coverage. Similarly, in inside-out detection systems, sensors can detect within any suitable area (e.g., 15-degree cone, 30-degree cone, etc.). Various areas are possible, and generally, higher tracking resolution can correspond to a greater number of detectable sensors / emitters with smaller beamwidths, and vice versa, as is apparent to a person skilled in the art with knowledge of this disclosure.

[0087] To illustrate this point, in Fig. Figure 7 shows several external tracking sensors 720(1-3). A 360-degree reference point is indicated, extending axially from the housing (e.g., lower part 712) and formed approximately around the center of the hexagonal ring, with a 0-degree reference positioned approximately perpendicular to the surface of the lower part with the three visible buttons. External sensors 720(1-3) (e.g., "Lighthouse" sensors) are shown at 300 degrees, 345 degrees, and 120 degrees, respectively, relative to the orientation of the pin 710 shown.

[0088] As from Fig.As can be seen in Figure 7, the geometry and design of the pen 710 are shown such that it can be detected primarily from an elevated angle at its sides. Some embodiments may use more sensors / emitters, including some that have been additionally added to the front section (lower part 712) and to the rear to improve tracking of the pen 712 directly from the front or directly from the rear, as may be the case with HMD tracking systems. Many variations, modifications, and alternative embodiments thereof are apparent to a person skilled in the art with knowledge of this disclosure.Although it has not been expressly mentioned for each of the embodiments described herein, it is of course possible to apply all the features described for a particular embodiment to any of the embodiments provided for in this disclosure, and it is to be assumed that a person skilled in the art, having knowledge of this disclosure, would know how these features can be mixed and combined in any desired combination.

[0089] In summary, the present invention discloses a pen device. In some embodiments, the pen device can be designed for use in an augmented reality / virtual reality (AR / VR) environment and can have a first part and a second part, wherein the first part of the housing can be substantially straight and designed to be held by hand by a user during use of the pen device. The first part can have a tip that serves as an interface between the input device and objects within the AR / VR environment. The second part of the housing can be curved and extend in three dimensions, including a section of the second part that is bent longitudinally to a line collinear with the first part of the housing.In some cases, the second part of the housing may have multiple emitters or sensors designed to support tracking of the pen device in the three-dimensional space of the AR / VR environment.

[0090] Each formulation of the form "at least one of A, B, or C" and the formulation "at least one of A, B, and C" use a disjunctive "or" and a disjunctive "and," respectively, such that these formulations, as used in this document, include any and all common and multiple permutations of A, B, C; that is, A alone, B alone, C alone, A and B in any order, A and C in any order, B and C in any order, and A, B, C in any order. More or fewer than three features may be used in these formulations.

[0091] In the claims, any reference numerals placed in parentheses are not to be interpreted as limiting the claim. The word "comprising" does not preclude the presence of elements or steps other than those listed in a claim. Furthermore, the terms "a" or "an," as used here, are to be understood as one or more than one. The use of introductory phrases such as "at least one" and "one or more" in the claims is also not to be interpreted as limiting a particular claim containing such an introduced claim element to inventions containing only one such element, even if the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an." The same applies to the use of definite articles.Unless otherwise specified, terms such as "first" and "second" are used arbitrarily to distinguish the elements described by these terms. Therefore, these terms are not necessarily intended to indicate any temporal or other prioritization of such elements. The mere fact that certain measures are cited in different claims does not imply that a combination of these measures cannot be used to one's advantage.

[0092] Unless expressly stated as incompatible, or unless physics or otherwise the embodiments, examples, or claims preclude such a combination, the features of the preceding embodiments and examples, as well as those of the following claims, may be combined in any suitable arrangement, particularly those in which a beneficial effect is achieved. This is not limited to a described benefit but may arise from an "ex post facto" advantage. This means that the combination of features is not limited by the described configurations, in particular the type (e.g., numbering) of the example(s), the embodiment(s), or the dependency of the claim(s).This also applies to the phrases "in one embodiment," "according to one embodiment," and the like, which are merely stylistic choices and are not to be interpreted as limiting the following features to a specific embodiment in all cases of the same or similar wording. That is to say, a reference to one or more embodiments can refer to one or more and / or all disclosed embodiments or combinations thereof. Likewise, a reference to "the" embodiment is not to be understood as being limited to the immediately preceding embodiment.

[0093] Certain figures in this text are flowcharts that illustrate methods and systems. It is noted that each block of the flowcharts, and combinations of blocks within the flowcharts, can be implemented by computer program instructions. These computer program instructions can be loaded into a computer or other programmable device to form a machine, such that the instructions executed on the computer or other programmable device create structures for implementing the functions specified in the block or blocks of the flowchart.The computer program instructions can also be stored in computer-readable memory and instruct a computer or other programmable device to operate in a specific way, such that the instructions stored in the computer-readable memory produce a product containing instruction structures that implement the function specified in the block or blocks of the flowchart. The computer program instructions can also be loaded into a computer or other programmable device so that a series of functional steps are executed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device implement steps to perform the functions specified in the block or blocks of the flowchart.Accordingly, blocks in flowcharts support combinations of structures for executing the specified functions and combinations of steps for executing the specified functions. Naturally, each block in the flowcharts and combinations of blocks in the flowcharts can be implemented by specialized hardware-based computer systems that execute the specified functions or steps, or combinations of specialized hardware and computer instructions.

[0094] Any computer programming language, such as C, C++, C# (CSharp), Perl, Ada, Python, Pascal, SmallTalk, FORTRAN, assembly language, and the like, can be used to implement machine instructions. Furthermore, depending on the requirements of the specific implementation, different programming approaches, such as procedural, object-oriented, or artificial intelligence techniques, can be employed. Compilers and / or virtual machine programs, executed by computer systems, generally translate higher-level programming languages ​​to generate sets of machine instructions that can be executed by one or more processors to perform a programmed function or set of functions.

[0095] The foregoing description of one or more implementations serves for illustration and explanation, but is not intended to be exhaustive or to limit the scope of the invention to the disclosed form. Modifications and variations are possible in light of the above teachings or can be derived from the implementation of different versions of the present disclosure.

Claims

[1] Pen device (110, 310(1-3), 410, 510, 610, 710) designed for use in an augmented / virtual reality (AR / VR) environment (100), wherein the pen device (110, 310(1-3), 410, 510, 610, 710) comprises: a housing that includes: a first part (312(1-3), 412, 512, 612, 712); and a second part (314(1-3), 414, 514, 614, 714); one or more processors (210) arranged in the housing; and a communication module (250) arranged in the housing and controlled by the one or more processors (210), wherein the communication module (250) is configured to establish a wireless electronic communication channel between the pen device (110, 310(1-3), 410, 510, 610, 710) and at least one host computer device (140), wherein the first part (312(1-3), 412, 512, 612, 712) of the housing is substantially straight and designed to be held by a user (102, 602) with one hand (104, 504, 604) during use of the pin device (110, 310(1-3), 410, 510, 610, 710), wherein the first part (312(1-3), 412, 512, 612, 712) comprises: an input element (416) designed to generate control signals in response to activation by the hand (104, 504, 604) of the user (102, 602); and a tip at one end of the first part (312(1-3), 412, 512, 612, 712) which is designed for handling as an interface between the pen device (110, 310(1-3), 410, 510, 610, 710) and objects within the AR / VR environment (100), wherein the second part (314(1-3), 414, 514, 614, 714) of the case is not straight and is curved in such a way that it extends in three dimensions, and wherein the second part (314(1-3), 414, 514, 614, 714) of the housing has several emitters or sensors (719) designed to support tracking of the pen device (110, 310(1-3), 410, 510, 610, 710) in the three-dimensional space of the AR / VR environment (100). [2] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 1, wherein a section of the second part (314(1-3), 414, 514, 614, 714) of the housing forms a polygonal ring. [3] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 2, wherein the second part (314(1-3), 414, 514, 614, 714) of the housing comprises a polygonal cross-section through at least a part of the polygonal ring. [4] Pin device (110, 310(1-3), 410, 510, 610, 710) according to one of claims 1 to 3, wherein the second part (314(1-3), 414, 514, 614, 714) of the housing has several planar facets (418, 718) and wherein the several emitters or sensors (719) are arranged on the several planar facets (418, 718). [5] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 4, wherein the multiple planar facets (418, 718) are each oriented such that they point in different directions of three-dimensional space. [6] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 4, wherein the multiple planar facets (418, 718) are oriented such that at least four of the planar facets (418, 718) are visible from any point 360 degrees around the pin device (110, 310(1-3), 410, 510, 610, 710). [7] Pin device (110, 310(1-3), 410, 510, 610, 710) according to any one of claims 1 to 6, wherein a center of gravity (616) of the pin device (110, 310(1-3), 410, 510, 610, 710) is located at the transition between the second part (314(1-3), 414, 514, 614, 714) and the first part (312(1-3), 412, 512, 612, 712). [8] Pin device (110, 310(1-3), 410, 510, 610, 710) according to any one of claims 1 to 7, wherein the pin device (110, 310(1-3), 410, 510, 610, 710) is balanced in the longitudinal and transverse directions at the transition between the second part (314(1-3), 414, 514, 614, 714) and the first part (312(1-3), 412, 512, 612, 712). [9] Pen device (110, 310(1-3), 410, 510, 610, 710) according to any one of claims 1 to 8, wherein the first part (312(1-3), 412, 512, 612, 712) of the housing is designed such that it can be held by the user (102, 602) with the hand (104, 504, 604) in a pen gripping position. [10] Pen device (110, 310(1-3), 410, 510, 610, 710) designed for use in an augmented / virtual reality (AR / VR) environment (100), wherein the pen device (110, 310(1-3), 410, 510, 610, 710) comprises: a case that features: a substantially straight first part (312(1-3), 412, 512, 612, 712) designed to be held by a user (102, 602) with one hand (104, 504, 604) during use of the pen device (110, 310(1-3), 410, 510, 610, 710); and a second part (314(1-3), 414, 514, 614, 714) which is not straight and is curved to extend in three dimensions, wherein the second part (314(1-3), 414, 514, 614, 714) of the housing has multiple emitters or sensors (719) configured to support tracking of the pen device (110, 310(1-3), 410, 510, 610, 710) in the three-dimensional space of the AR / VR environment (100), wherein a center of gravity (616) of the pen device (110, 310(1-3), 410, 510, 610, 710) is located at the transition between the second part (314(1-3), 414, 514, 614, 714) and the first part (312(1-3), 412, 512, 612, 712) such that the pen device (110, 310(1-3), 410, 510, 610, 710) is configured such that, when the hand (104, 504, 604) of the user (102, 602) is aligned in a pen device insertion position, it is located in a space between the index finger and thumb (606) of the hand (104, 504, 604) of the user (102, 602) It can be balanced in a self-supporting manner. [11] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 10, which further comprises a section of the second part (312(1-3), 412, 512, 612, 712) which is bent longitudinally in the direction of a line collinear with the first part (314(1-3), 414, 514, 614, 714) of the housing. [12] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 11, wherein a section of the second part (314(1-3), 414, 514, 614, 714) of the housing forms a hexagonal ring and the housing has a hexagonal cross-section through at least a part of the hexagonal ring. [13] Pin device (110, 310(1-3), 410, 510, 610, 710) according to one of claims 10 to 12, wherein the second part (314(1-3), 414, 514, 614, 714) of the housing has several planar facets (418, 718) and wherein the several emitters or sensors (719) are arranged on the several planar facets (418, 718). [14] Pin device (110, 310(1-3), 410, 510, 610, 710) according to claim 13, wherein the multiple planar facets (418, 718) are each oriented such that they point in different directions of three-dimensional space. [15] Pen device (110, 310(1-3), 410, 510, 610, 710) according to any one of claims 10 to 14, wherein the multiple planar facets (418, 718) are oriented such that at least four of the multiple planar facets (418, 718) are visible from any point 360 degrees around the pen device (110, 310(1-3), 410, 510, 610, 710). [16] Pin device (110, 310(1-3), 410, 510, 610, 710) according to any one of claims 10 to 15, wherein the pin device (110, 310(1-3), 410, 510, 610, 710) is longitudinally and transversely balanced at the transition between the second part (314(1-3), 414, 514, 614, 714) and the first part (312(1-3), 412, 512, 612, 712). [17] Input device (110, 310(1-3), 410, 510, 610, 710), wherein the input device (110, 310(1-3), 410, 510, 610, 710) is designed for use in an augmented / virtual reality (AR / VR) environment (100) and wherein the input device (110, 310(1-3), 410, 510, 610, 710) comprises: a case that features: a first part (312(1-3), 412, 512, 612, 712); and a second part (314(1-3), 414, 514, 614, 714); wherein the first part (312(1-3), 412, 512, 612, 712) of the housing is designed to be held by a user (102, 602) with one hand (104, 504, 604) during use of the input device (110, 310(1-3), 410, 510, 610, 710), wherein the second part (314(1-3), 414, 514, 614, 714) of the case is not straight and is curved in such a way that it extends in three dimensions, and has a section forming a hexagonal ring with a hexagonal cross-section, wherein the hexagonal ring has a section that is bent longitudinally in the direction of a line collinear with the first part (312(1-3), 412, 512, 612, 712) of the casing, and wherein the hexagonal ring has several planar facets (418, 718) wherein several emitters or sensors (719) are arranged on the several planar facets (418, 718) which are designed to support tracking of the input device (110, 310(1-3), 410, 510, 610, 710) in the three-dimensional space of the AR / VR environment (100). [18] Input device (110, 310(1-3), 410, 510, 610, 710) according to claim 17, wherein the multiple planar facets (418, 718) are each oriented such that they point in different directions of three-dimensional space, and wherein each of the multiple planar facets (418, 718) is oriented such that at least four of the planar facets (418, 718) are visible from any point 360 degrees around the input device (110, 310(1-3), 410, 510, 610, 710).

Images