Computerized system control components, methods, and non-transitory computer readable media

Abstract

Computerized system control components, methods, and non-transitory computer-readable media. Recent technological advances have expanded the breadth, scope, and affordability of virtual reality (VR) and augmented reality (AR) systems. VR and AR systems are accessible to more people than ever before. However, handheld physical controls for these systems have not made similar advances. As a result, it remains difficult for developers to create applications that exploit the full range of human capabilities. In this proposal, the applicant describes a system and method for combining an off-the-shelf smartwatch with a set of inexpensive handles or sheaths to control a VR / AR system. Using the applicant's method, the controller's computation and power are all derived from the smartwatch unit, allowing the handle to take almost any form and allowing developers and designers to integrate more diverse interaction styles into their applications.

Description

Technical Field

[0001] The disclosed embodiments generally relate to virtual reality (VR) and augmented reality (AR) systems, and more particularly to a low-power personalized smart controller for VR / AR interaction. Background Art

[0002] The market for VR and AR continues to expand: its total market size is expected to expand from $27 billion in 2018 to $209.2 billion in 2022. Recent technological advances have expanded the breadth, scope, and affordability of virtual reality (VR) and augmented reality (AR) systems. More people than ever before have access to VR and AR systems. However, the expressiveness of interactive controls has not kept pace with this expansion of the VR / AR application market. VR / AR systems often use proprietary controllers that leave little room for developers to experiment and respond to the full range of human capabilities, such as Figure 1a , Figure 1b , Figure 1c and Figure 1d Some of the dedicated controllers designed for specific systems available on the market include the Vive controller for the HTC Vive system, the Oculus Touch controller for the Oculus Rift, the Daydream controller for the Daydream View available from Google, and the Manus VR Gloves controller for Manus VR. For the above reasons, it is still difficult for developers to create applications that utilize the full range of human capabilities.

[0003] Therefore, in view of the above and other shortcomings of conventional technology, there is a need for new and improved handheld physical controls for VR and AR systems that provide developers with the necessary flexibility to develop new user applications. Summary of the invention

[0004] Implementations described herein are directed to systems and methods that substantially eliminate one or more of the above and other problems associated with conventional VR / AR control systems.

[0005] According to one aspect of the embodiments described herein, there is provided a computerized system control assembly comprising: a holding portion configured to be held by a user, the holding portion including at least one control component; a mobile computing device comprising a processing unit, a memory and at least one sensor, the mobile computing device being attached to the holding portion, wherein the memory stores a plurality of executable computer instructions for allowing two-way communication between the mobile computing device and an external computer system and for sending sensor readings from the at least one sensor to the external computer system and receiving feedback from the external computer system and providing the received feedback to the user, and wherein the mobile computing device operates in conjunction with the at least one control component.

[0006] In one or more implementations, the external computer system is an augmented reality system.

[0007] In one or more embodiments, the external computer system is a virtual reality system.

[0008] In one or more implementations, the mobile computing device is a smartwatch.

[0009] In one or more embodiments, the smart watch is attached to the grip portion via a strap.

[0010] In one or more embodiments, the smart watch is attached to the grip portion by magnetic attachment with the watch band removed.

[0011] In one or more embodiments, the at least one control component is a magnetic trigger.

[0012] In one or more embodiments, the magnetic trigger is configured to be activated by a user's finger.

[0013] In one or more embodiments, the at least one sensor is a magnetometer configured to sense activation of the magnetic trigger by a user.

[0014] In one or more embodiments, the grip portion further comprises a sensor input migration component for migrating the input of the at least one sensor from the mobile computing device to the grip portion.

[0015] In one or more embodiments, the at least one sensor is a heart rate sensor, and wherein the input migration component is an optical fiber optically coupled to the heart rate sensor.

[0016] In one or more embodiments, the mobile computing device communicates with the at least one control component via near field communication (NFC).

[0017] In one or more embodiments, the at least one control component is an actuator.

[0018] In one or more embodiments, the at least one control component is a button.

[0019] In one or more embodiments, the at least one control component is a slider.

[0020] In one or more embodiments, the at least one control component is powered by the mobile computing device via near field communication (NFC).

[0021] In one or more embodiments, the grip portion further comprises at least one light source for tracking the position or orientation of the grip portion.

[0022] In one or more embodiments, the grip portion further comprises at least one light detector for tracking the position or orientation of the grip portion.

[0023] According to another aspect of the embodiments described herein, a method is provided, the method involving: attaching a grip portion to a mobile computing device, the grip portion being configured to be gripped by a user, the grip portion comprising at least one control component, the mobile computing device comprising a processing unit, a memory, and at least one sensor; allowing two-way communication between the mobile computing device and an external computer system; sending sensor readings from the at least one sensor to the external computer system; receiving feedback from the external computer system; and providing the received feedback to the user, wherein the mobile computing device operates in conjunction with the at least one control component.

[0024] According to another aspect of the implementation described herein, a non-transitory computer-readable medium that specifically implements an instruction set is provided, wherein the instruction set implements a method performed in conjunction with a computerized system control component, the computerized system control component comprising a holding portion and a mobile computing device, the holding portion being configured to be held by a user, the holding portion comprising at least one control component, the mobile computing device comprising a processing unit, a non-transitory computer-readable medium, and at least one sensor, the mobile computing device being attached to the holding portion; the method involving: allowing two-way communication between the mobile computing device and an external computer system; sending sensor readings from the at least one sensor to the external computer system; receiving feedback from the external computer system; and providing the received feedback to the user, wherein the mobile computing device operates in conjunction with the at least one control component.

[0025] Additional aspects related to the present invention will be explained in part in the following description, and in part will be obvious from the description, or can be learned through practice of the present invention. Aspects of the present invention can be realized and obtained by elements and combinations of various elements and aspects particularly pointed out in the following detailed description and the appended claims.

[0026] It will be understood that both the foregoing and the following description are exemplary and illustrative only and are not intended to limit the claimed invention or its applications in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present invention and, together with the specification, are used to explain and illustrate the principles of the technology of the present invention. Specifically:

[0028] Figure 1a A commercially available Vive controller is shown.

[0029] Figure 1b A commercially available Opus Rift controller is shown.

[0030] Figure 1c A Daydream controller available from Google is shown.

[0031] Figure 1d Commercially available Manus VR Gloves controllers are shown.

[0032] Figure 2 An exemplary embodiment of a system including a low-power personalized smart controller for VR / AR interaction is shown.

[0033] Figure 3 An exemplary operational sequence of an embodiment of a system including a low-power personalized smart controller for VR / AR interaction is shown.

[0034] Figure 4a , Figure 4b , Figure 4c and Figure 4d Various embodiments of a low-power personalized smart handle for VR / AR interaction are shown.

[0035] Figure 5 An exemplary embodiment of a computer system that can be used to implement the inventive techniques described herein is shown. DETAILED DESCRIPTION

[0036] In the following detailed description, reference will be made to the accompanying drawings, in which the same functional elements are referred to by the same reference numerals. The above-mentioned drawings show specific embodiments and implementations that conform to the principles of the present invention as examples rather than as limitations. These implementations are described in sufficient detail to enable those skilled in the art to practice the present invention, and it will be understood that other implementations may be used and that structural changes and / or replacements of various elements may be made without departing from the scope and spirit of the present invention. Therefore, the following detailed description should not be interpreted in a limiting sense. In addition, the various embodiments of the present invention described may be implemented in the form of software running on a general-purpose computer, in the form of specialized hardware, or in a combination of software and hardware.

[0037] According to one aspect of the embodiments described herein, a system and method for implementing a low-power personalized smart handle for VR / AR interaction is provided. One embodiment combines a user's personal smartwatch with an inexpensive handle or sheath. In various embodiments, the described smartwatch can be combined with the handle directly (leaving the strap) or by removing the watch unit from its strap and integrating it into the handle. Using this method, developers will build smartwatch software applications that can be used in combination with a series of separate handles to control VR / AR applications. This method has multiple advantages over the prior art and other alternative methods.

[0038] In one or more embodiments, by separating the computing unit from the physical handle, developers can integrate a more diverse controller design, allowing them to customize the user experience for specific VR / AR applications. In addition, this approach allows designers and developers to more easily personalize the experience of each application to match the user preferences imported from the smartwatch. Developers can also release multiple different inexpensive handles to provide users with different ways to experience their applications, or support handles for non-conventional (e.g., disabled) users. Finally, the handle itself does not require a separate power supply: the applicant uses the smartwatch to power not only the handle buttons, but also other sensors of the smartwatch (e.g., a heart rate sensor).

[0039] In contrast, conventional controllers do not provide any way to integrate different handle styles, personalize their use, or integrate external sensors. Although applicants are not aware of any universal controller for VR / AR systems, because there is no universal physical form, nor an agreed-upon set of interactors or sensors to be built into such a unit, it is impossible to construct a controller that provides all of the benefits of applicants' system.

[0040] In various embodiments, the described systems and methods combine a smartwatch with a set of inexpensive controllers or cases to control various VR / AR systems. In various embodiments, the watch and controllers are connected using one of two methods. According to the first method, the connection involves attaching the watch and its strap to the controller. According to the second method, the attachment involves detaching the watch from its strap and integrating the watch so detached into the controller.

[0041] In one or more embodiments, the handle contains various unpowered mechanical elements, including but not limited to magnetized triggers, buttons, sliders, LEDs, and other sensors that can be shaped or customized for specific VR / AR applications. In one or more embodiments, the handle is designed to utilize the NFC of a smartwatch to power its onboard mechanical elements. In one or more embodiments, a software application deployed on a smartwatch can convert events sensed from the handle into events related to the VR / AR application to which it is connected.

[0042] In one or more embodiments, the system 200 is shown in Figure 2 and described in detail below. The system described herein operates in conjunction with a VR / AR system 201, which is connected to the described system 200 via Bluetooth 205 or other suitable wireless data interconnect. Those of ordinary skill in the art will understand that the exact nature of the data interconnect is not critical to the inventive concepts described herein. The customizable software application 204 is used to communicate wirelessly with a computing device running the VR / AR system 201. In various embodiments, such a system can be a computer, a smart phone, or a similar computing device. In various embodiments, the user downloads the customizable software application 204 to their smart watch 202.

[0043] In one or more embodiments, the system 200 includes a smart watch 202 and a low-power smart handle 203. The smart watch 202 may include an inertial measurement unit 206, which is an electronic device that uses a combination of an accelerometer and a gyroscope (and sometimes a magnetometer) to measure and report the subject's specific force, angular velocity, and sometimes the magnetic field around the subject. In addition, the smart watch 202 may include a certain number of physiological sensors, such as heart rate, blood pressure, blood oxygen level, blood glucose level sensors, and others.

[0044] In one or more embodiments, a touch screen sensor 208, a microphone 209, a screen 210, a haptic actuator 211, an audio speaker 212, and a near field communication (NFC) controller 213 may also be incorporated into the smart watch 202. The design, manufacture, and operation of all of the above units are well known in the art and are widely available in the public literature.

[0045] In one or more embodiments, the smart watch 202 and Figure 2 In various embodiments, the handle may include one or more mechanical triggers 215, optical fibers 216, one or more actuators 217, an NFC controller 218, and a low-power display 219. A person of ordinary skill in the art will appreciate that in various embodiments, Figure 2 The low-power intelligent handle indicated by numeral 203 may include other suitable components and sensors. Therefore, the present invention is not limited to the components listed above.

[0046] Figure 3 An exemplary operational sequence 300 of an embodiment of the described system 200 is shown. In one or more embodiments, first, the user removes the smart watch 202 (see Figure 3 As is well known in the art, the watch is secured to the user's hand using a strap. If the handle 203 is not attached using a strap ( Figure 3 302 in the figure), the user removes the watch band from the smart watch 202 (see step 303). Subsequently, the user attaches the smart watch 202 to the handle 203 with or without the watch band (see step 304).

[0047] In one or more embodiments, at step 305, the user downloads and launches the customizable smart watch application 204. The smart watch 202 uses the customizable smart watch application 204 to wirelessly communicate with the computing device (which can be a computer, smart phone, or any other computing device) running the VR / AR system 201. If the AR / VR control system requires calibration (step 306), the user uses the instructions provided by the customizable smart watch application 204 to perform the calibration of the system (see step 307). Finally, at step 308, data is sent between the handle 203, the customizable smart watch application 204, and the AR / VR system 201.

[0048] In one or more embodiments, the smart watch 202 can be attached to the low power smart handle 203 using one of the three methods described in detail below. In one embodiment 400, the smart watch 401 can be attached to the low power smart handle 403 by wrapping the watch with its strap 402 around a dedicated groove in the handle 403 (see Figure 4a ). The user holds the handle 403 and the smart watch 401 is in his or her hand (wrist) 404.

[0049] In another embodiment, the smart watch 401 is attached to the handle 411 by removing the smart watch 401 from its strap (not shown) and attaching the smart watch 401 to a compatible electrical connector on the handle 411, such as Figure 4b As shown ( Figure 4b In one or more embodiments, the handle 411 may be equipped with a mechanical trigger 412. The above-mentioned compatible electrical connector on the handle 411 may include a plurality of spring-loaded electrical connection pins disposed in dedicated slots provided on the handle for the watch 401.

[0050] In another embodiment, the smart watch is attached to the handle by removing the watch from its strap and attaching it to the device using a magnetic connector, such as Figure 4c and Figure 4d Specifically, Figure 4c As shown, the smart watch 401 is removed from the strap and attached to the handle 421 using magnets provided on the back of the smart watch and on the handle 421. The aforementioned magnets operate to keep the watch in place on the handle 421. To facilitate the fixing of the smart watch 401 to the handle 421, the latter may be equipped with a special slot. Figure 4c As shown, the smart handle 421 may further include one or more buttons 423 and one or more indicators (photodiodes) 422. Figure 4d As shown, the smart watch 401 is removed from the watchband and attached to a spherical AR / VR control 432 having the shape of a ball.

[0051] In another embodiment, once the smartwatch is attached to the smart handle, the user launches a customizable smartwatch application 204 running on the smartwatch 202. The customizable smartwatch application 204 wirelessly connects the smartwatch 202 to the AR / VR system 201 and begins sending one or more of the following data records to the VR system: (1) IMU 206 sensor data, including but not limited to accelerometer and magnetometer data; (2) touch events from the smartwatch's touch screen 210; and (3) the user's heart rate, blood oxygen, etc. readings and all trigger presses obtained using physiological sensors 207. These are classified by the application based on changes in magnetometer readings (see "Unpowered Triggers using Magnetic Motion" below). In addition, the smartwatch 202 can send to the AR / VR system 201 (4) sensor readings from low-power input elements such as buttons, sliders, etc., transmitted via the watch's NFC, as described in detail below.

[0052] In one or more embodiments, the customizable smart watch application 204 may receive the following data from the VR / AR system 201: (1) a system call to actuate the watch's vibration motor; and (2) a system call to apply changes to low-power display elements on the device, for example using the watch's NFC unit; (3) a system call to illuminate the smart watch's screen 210 (for AR use cases).

[0053] In one or more embodiments, if this is the first time the user is running an application using a particular controller, the user will go through the following system training / calibration steps: (1) the customizable smart watch application 204 will ask the user to hold the controller in different orientations and perform air gestures; (2) the customizable smart watch application 204 will ask the user to swipe up, down, left, and right on the watch's touch screen; (3) the customizable smart watch application 204 will ask the user to demonstrate different triggers and map them to known events (e.g., pressing the main trigger immediately or tapping the watch's screen to skip).

[0054] One embodiment utilizes a smartwatch IMU 206 to support six degrees of freedom interaction between the combined smartwatch-handle system and the VR / AR system 201. The above system uses the smartwatch's touch screen 210 for touch input and the smartwatch's vibrotactile actuator 211 for tactile feedback. This embodiment can also utilize the smartwatch's heart rate sensor (physiological sensor) 207 by using optical fibers to "migrate" the sensing position. In addition, in one embodiment, the unpowered mechanical element (e.g., trigger 215) contains a magnet, and the smartwatch can respond based on changes in the smartwatch's magnetometer readings ( Figure 2 In one embodiment, additional I / O elements (e.g., buttons, sliders, and LEDs) may also be included that communicate with the smartwatch via NFC. In one embodiment, these elements may be powered via NFC, and thus the handle 203 (despite containing electronic components) never needs to be charged.

[0055] In one or more embodiments, two different methods are used to combine the smart watch 202 and the associated handle 203. First, the smart watch and its strap are assembled to the device (see Figure 4a ). Figure 4a In the embodiment, the smart watch 401 and the watch band 402 are mounted perpendicular to the wrist 404. Alternatively, the smart watch 401 and the watch band 402 can also be mounted parallel to the wrist 404. In the second method, the smart watch is first removed from its watch band and mounted into the device, see Figure 4b , Figure 4c and Figure 4d In a second approach, the device attaches to a mechanical interface of the watch for a custom strap, or uses a magnetic connection. These magnetic connections do not interfere with the sensing of the magnetic trigger 412; however, additional calibration is required in this embodiment to establish a baseline to compare against the trigger event.

[0056] Those of ordinary skill in the art will appreciate that most smart watches today include optical heart rate (HR) sensors. In the context of VR / HR systems, heart rate is used, for example, to control the flow of a game, as described in Araki, Hayato et al., "Development of a Horror Game that Route Branches by the Player's Pulse Rate" (Proceedings of IIUII Companion 2018). For example, a system may control how scary a VR game is based on the player's heart rate. Previous studies have found a relationship between mental stress and changes in components of heart rate variability (HRV). The challenge in the case of the described embodiments is that smart watches with HR sensors will have sensors on the bottom side of the watch so that they sense HR through the wrist; however, in at least some of the described embodiments, the smart watch is not worn on the wrist. In order to still be able to use the HR sensor of the watch, in one embodiment, the system optically utilizes these worn HR sensors and uses optical fibers to migrate the sensing position of the HR sensor of the watch.

[0057] Specifically, photoplethysmography (PPG) heart rate sensors use a combination of one or more LED emitters and a light sensor to detect blood changes in capillaries, as described, for example, in U.S. Patent Nos. 3,228,391, 3,608,545, and 8,948,832. Figure 1a to Figure 1d In the embodiment of the described system shown, an optical fiber 221 embedded in the device allows light to be emitted and reflected to be captured under the user's finger (or palm). It should be noted that in order for this method to work, the smart watch's sensor must be precisely aligned over the optical fiber. This can be achieved in a variety of ways, including physical ways (a "slot" in the smart watch prevents misalignment) and magnetic ways (using magnets to "snap" the watch into place).

[0058] One of ordinary skill in the art will appreciate that most modern smart watches include a magnetometer. While this sensor is most commonly used in navigation applications, it can also be used to sense unpowered input by manipulating a magnet near the smart watch. Prior work by Chris Harrison and Scott E. Hudson. 2009. Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices (Proceedings of UIST '09) demonstrated the use of a ring with a magnet to recognize gestures performed around a watch. The first version of Google Cardboard, described in U.S. Patent No. D750074S1, included magnets to replace simple buttons. One embodiment of the smart handle similarly includes a physical unpowered trigger (see Figure 4b 412 in FIG. 4 ). Each trigger mechanism includes a magnet, the movement of which can be sensed by the smart watch. For example, pressing the trigger can cause the magnet to spin off-axis, thereby generating a recognizable pattern.

[0059] Those of ordinary skill in the art will appreciate that most modern smart watches have vibration motors embedded in them. One embodiment of a smart controller utilizes this form of output to provide vibrating tactile feedback for virtual or augmented reality. The VR / AR system 201 can send messages to the controller application 204 running on the smart watch based on elements in the virtual / augmented environment, causing it to vibrate. In one embodiment, the VR / AR system 201 can instruct the smart watch 202 to use the tactile actuator 211 to generate a short vibration to indicate to the user that it is hovering over a graspable virtual object.

[0060] In one or more embodiments, a controller application running on a smartwatch may provide vibrotactile feedback based on the parameters of the interaction, agnostic to the VR / AR environment itself. For example, in response to a movement or 3D gesture performed by the controller (and sensed by the watch's IMU 206). Or, for example, if touch interactions on the watch are treated as continuously acquired interactions (similar to a joystick), where dragging a finger away from the center of the watch results in faster movement, vibrotactile feedback may be used to indicate to the user how far the finger is from the center point.

[0061] Those of ordinary skill in the art will appreciate that most modern smart watches are equipped with near field communication (commonly referred to as NFC) capabilities. Although this is most commonly used for smart payments (Apple Pay, Android Pay, etc.), NFC allows data and power transfer between various devices. In any of the above embodiments, the smart watch 202 can power and communicate with the smart handle device (handle) 203. The smart handle may include additional inputs (e.g., buttons, sliders, capacitive touch sensors, etc.), additional outputs (e.g., LEDs, displays, additional vibration motors, speakers, etc.), or a combination of both (see Figure 4c and Figure 4d ). In one or more embodiments, these I / O elements are powered via NFC and communicate with the smart watch 202 via NFC 220. While tactile elements such as buttons and additional vibration motors are suitable for both VR and AR, visual elements are more suitable for AR applications where the user can physically look at the device as well as the smart watch screen.

[0062] In one or more embodiments, one of the outputs may be an LED for external tracking (see Figure 4c 422 in FIG. 4 ). The AR / VR system 201 can use these LEDs to locate the controller in 3D space, just as it would when using its dedicated controller. Alternatively, one of these inputs can be a photodiode similarly used for position tracking, where the AR / VR system 201 emits light that is picked up by these photodiodes 422, similar to the operating mode of traditional AR / VR controllers known in the art.

[0063] Exemplary Implementation of a Computer System

[0064] Figure 5 An exemplary embodiment of a computer 500 that can be used to implement the techniques described herein, such as an AR / VR system 201, is shown. In one or more embodiments, the computer 500 can be implemented within the form factor of a mobile computing device familiar to those skilled in the art. In an alternative embodiment, the computer 500 can be implemented based on a laptop or notebook computer. However, in an alternative embodiment, the computer 500 can be a professional computing system.

[0065] The computer 500 may include a data bus 504 or other interconnection or communication mechanism for transferring information across the various hardware components of the computer 500 and a central processing unit (CPU or simply processor) 501 coupled to the data bus 504 for processing information and performing other computing and control tasks. The computer 500 also includes a memory 512 (e.g., a random access memory (RAM) or other dynamic storage device) coupled to the data bus 504 for storing various information and instructions to be executed by the processor 501. The memory 512 may also include a persistent storage device such as a magnetic disk, an optical disk, a solid-state flash memory device, or other non-volatile solid-state storage device.

[0066] In one or more embodiments, the memory 512 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processor 501. Optionally, the computer 500 may also include a read-only memory (ROM or EPROM) 502 or other static storage device, which is connected to the data bus 504 to store static information and instructions of the processor 501, such as firmware required for the operation of the computer 500, a basic input and output system (BIOS), and various configuration parameters of the computer 500.

[0067] In one or more embodiments, the computer 500 may further include a camera 510 for capturing still images and real-time video. In addition, the computer 500 may include a microphone 511 for picking up audio and a microphone control module 509 for automatically muting and unmuting the microphone 511 based on the user's head posture.

[0068] In one or more embodiments, the computer 500 may further include a communication interface, such as a network interface 505 coupled to the data bus 504. The network interface 505 may be configured to establish a connection between the computer 500 and the Internet 524 using at least one of a WIFI interface 507 and a cellular network (GSM or CDMA) adapter 508. The network interface 505 may be configured to provide two-way data communication between the computer 500 and the Internet 524. The WIFI interface 507 may operate in accordance with 802.11a, 802.11b, 802.11g and / or 802.11n protocols and Bluetooth protocols well known to those of ordinary skill in the art. In an exemplary implementation, the WIFI interface 507 and the cellular network (GSM or CDMA) adapter 508 send and receive electrical or electromagnetic signals that carry digital data streams representing various types of information.

[0069] In one or more embodiments, the Internet 524 provides data communication to other network resources, usually through one or more sub-networks. Therefore, the computer 500 can access various network resources located anywhere on the Internet 524, such as remote media servers, web servers, other content servers, and other network data storage resources. In one or more embodiments, the computer 500 is configured to send and receive messages, media, and other data (including application code) through various networks including the Internet 524 with the help of the network interface 505. In the Internet example, when the computer 500 acts as a network client, it can request the code or data of the application program executed in the computer 500. Similarly, it can send various data or computer codes to other network resources.

[0070] In one or more embodiments, the functions described herein are implemented by the computer 500 in response to the processor 501 executing one or more sequences of one or more instructions contained in the memory 512. These instructions can be read into the memory 512 from another computer-readable medium. The execution of the sequence of instructions contained in the memory 512 causes the processor 501 to perform the various processing steps described herein. In alternative embodiments, instead of software instructions or in combination with software instructions, hard-wired circuits can be used to implement embodiments of the present invention. Therefore, embodiments of the present invention are not limited to any specific combination of hardware circuits and software.

[0071] The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to the processor 501 for execution. Computer-readable media is merely one example of a machine-readable medium that may carry instructions for implementing any of the methods and / or techniques described herein. Such media may take many forms, including but not limited to non-volatile media and volatile media.

[0072] Common forms of non-transitory computer-readable media include, for example, floppy disks, floppy disks, hard disks, tapes or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs, EPROMs, FLASH-EPROMs, flash drives, memory cards, any other memory chips or boxes, or any other media that can be read by a computer. Various forms of computer-readable media may involve carrying one or more sequences of one or more instructions to processor 501 for execution. For example, instructions may be initially carried on a disk from a remote computer. Alternatively, a remote computer may load instructions into its dynamic memory and send instructions via the Internet 524. Specifically, computer instructions may be downloaded from the above-mentioned remote computer to the memory 512 of computer 500 via the Internet 4 using various network data communication protocols well known in the art.

[0073] In one or more embodiments, the memory 512 of the computer 500 may store any of the following software programs, applications, and / or modules:

[0074] 1. Operating system (OS) 513, which may be a mobile operating system for implementing basic system services and managing various hardware components of computer 500. Exemplary implementations of operating system 513 are well known to those skilled in the art, and may include any mobile operating system now known or later developed. In addition, a network communication module 514 may be provided for allowing network communication using network interface 505.

[0075] 2. The software module 515 may include, for example, a set of software modules executed by the processor 501 of the computer 500, which enables the computer 500 to perform certain predetermined functions, such as communicating with the smart watch 202 (module 517) and / or operating the AR / VR system (module 516).

[0076] 3. The data storage device 518 can be used to store various parameters and thresholds 519, for example.

[0077] Finally, it should be understood that the processing and techniques described herein are not inherently related to any particular device and can be implemented by any suitable combination of components. In addition, according to the teachings described herein, various types of general-purpose devices can be used. It can also be proved that it is advantageous to construct professional equipment to perform the method steps described herein. The present invention is described with respect to specific examples, which are intended to be illustrative and non-restrictive in all aspects. It will be appreciated by those skilled in the art that many different combinations of hardware, software and firmware will be suitable for practicing the present invention. For example, the described software can be implemented with various programming or scripting languages, such as assembler, C / C++, Objective-C, peri, shell, PHP, Java and any program or scripting language now known or developed later.

[0078] In addition, considering the description and practice of the invention disclosed herein, other implementations of the invention will be apparent to those skilled in the art. Various aspects and / or components of the described embodiments may be used alone or in any combination in a system and method for implementing a low-power personalized smart handle for VR / AR interaction. The description and examples are intended to be considered merely exemplary, and the true scope and spirit of the invention are indicated by the appended claims.

Claims

1. A computerized system control component, the computerized system control component comprising: a. a grip portion, the grip portion being configured to be held by a user's hand, the grip portion comprising at least one control component; as well as b. a mobile computing device comprising a processing unit, a memory, and at least one sensor, the mobile computing device being removably attached to the holding portion, wherein the memory stores a plurality of executable computer instructions to: allowing two-way communication between the mobile computing device and an external computer system, receiving user input on the at least one control component of the grip portion via near field communication (NFC) and sending the user input to the external computer system, sending sensor readings from the at least one sensor to the external computer system, and receiving feedback from the external computer system and providing the received feedback to the user, and wherein the mobile computing device operates in conjunction with the at least one control component, wherein the grip portion utilizes the output of a vibration motor embedded in the mobile computing device to provide vibrotactile feedback for virtual reality or augmented reality, and wherein if the touch interaction on the mobile computing device is treated as a continuously acquired interaction, the vibrotactile feedback is used to indicate to the user the distance of the finger from a center point of the mobile computing device, wherein in the continuously acquired interaction, dragging the finger away from the center point of the mobile computing device results in faster movement.

2. The computerized system control assembly of claim 1, wherein: The external computer system is an augmented reality system.

3. The computerized system control assembly of claim 1, wherein: The external computer system is a virtual reality system.

4. The computerized system control assembly of claim 1, wherein: The mobile computing device is a smart watch.

5. The computerized system control assembly of claim 4, wherein: The smart watch is attached to the grip portion via a strap.

6. The computerized system control assembly of claim 4, wherein: The smartwatch is attached to the grip portion by magnetic attachment with the watch strap removed.

7. The computerized system control assembly of claim 1, wherein: The at least one control component is a magnetic trigger.

8. The computerized system control assembly of claim 7, wherein: The magnetic trigger is configured to be activated by a user's finger.

9. The computerized system control assembly of claim 7, wherein: The at least one sensor is a magnetometer configured to sense activation of the magnetic trigger by the user.

10. The computerized system control assembly of claim 1, wherein: The grip portion also includes a sensor input migration component for migrating input of the at least one sensor from the mobile computing device to the grip portion.

11. The computerized system control assembly of claim 10, wherein: The at least one sensor is a heart rate sensor, and wherein the input migration component is an optical fiber optically coupled to the heart rate sensor.

12. The computerized system control assembly of claim 1, wherein: The mobile computing device communicates with the at least one control component via the NFC.

13. The computerized system control assembly of claim 1, wherein: The at least one control component is an actuator.

14. The computerized system control assembly of claim 1, wherein: The at least one control component is a button.

15. The computerized system control assembly of claim 1, wherein: The at least one control component is a slider.

16. The computerized system control assembly of claim 1, wherein: The at least one control component is powered by the mobile computing device via the NFC.

17. The computerized system control assembly of claim 1, wherein: The grip portion also includes at least one light source for tracking the position or orientation of the grip portion.

18. The computerized system control assembly of claim 1, wherein: The grip portion also includes at least one light detector for tracking the position or orientation of the grip portion.

19. The computerized system control assembly of claim 1, wherein: The external computer system is an augmented reality system or a virtual reality system, and the mobile computing device is a smart watch, and wherein the smart watch is attached to the attachment surface of the grip portion by means of one or more connectors with the watch band removed.

20. A method performed by a computerized system control component as claimed in any one of claims 1 to 18, the method comprising the steps of: a. attaching a grip portion to a mobile computing device, the grip portion being configured to be held by a user's hand, the grip portion comprising at least one control component, the mobile computing device comprising a processing unit, a memory, and at least one sensor; b. Allowing two-way communication between the mobile computing device and an external computer system; c. receiving user input on at least one control component of the holding portion via near field communication NFC, and sending the user input to the external computer system; d. sending sensor readings from said at least one sensor to said external computer system; e. receiving feedback from the external computer system; as well as f. providing the received feedback to the user, wherein the mobile computing device operates in conjunction with the at least one control component, wherein the grip portion utilizes the output of a vibration motor embedded in the mobile computing device to provide vibrotactile feedback for virtual reality or augmented reality, and wherein if the touch interaction on the mobile computing device is treated as a continuously acquired interaction, the vibrotactile feedback is used to indicate to the user the distance of the finger from a center point of the mobile computing device, wherein in the continuously acquired interaction, dragging the finger away from the center point of the mobile computing device results in faster movement.

21. A non-transitory computer readable medium embodying a set of instructions implementing a method performed in conjunction with a computerized system control assembly, the computerized system control assembly comprising a grip portion and a mobile computing device, the grip portion being configured to be gripped by a hand of a user, the grip portion comprising at least one control component, the mobile computing device comprising a processing unit, the non-transitory computer readable medium, and at least one sensor, the mobile computing device being attached to the grip portion; The method comprises the following steps: a. Allowing two-way communication between the mobile computing device and an external computer system; b. receiving user input on the at least one control component of the holding portion via near field communication NFC, and sending the user input to the external computer system, c. sending sensor readings from said at least one sensor to said external computer system; d. receiving feedback from the external computer system; and e. providing the received feedback to the user, wherein the mobile computing device operates in conjunction with the at least one control component, wherein the grip portion utilizes the output of a vibration motor embedded in the mobile computing device to provide vibrotactile feedback for virtual reality or augmented reality, and wherein if the touch interaction on the mobile computing device is treated as a continuously acquired interaction, the vibrotactile feedback is used to indicate to the user the distance of the finger from a center point of the mobile computing device, wherein in the continuously acquired interaction, dragging the finger away from the center point of the mobile computing device results in faster movement.

Images