Foldable container for XR glasses
By designing a foldable and expandable protective case, the input range and computing power of lightweight glasses are enhanced, and the limited performance of lightweight glasses is solved, achieving a more convenient user experience and device carrying.
Patent Information
- Application Number
- CN202280093815.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-29
AI Technical Summary
Lightweight extended reality glasses are limited in performance due to reduced size and weight, lacking sufficient input range and computing power, making it inconvenient to carry peripheral devices.
Designed a foldable and expandable protective housing or container with built-in processors, communication links and input devices to enhance the input range and computing power of wearable XR displays, including face capture cameras, tactile input surfaces and extendable Input surface, automatically reconfigure the operation mode to match user input.
Enables lightweight wearable XR displays to provide comprehensive input range and computing power like larger and heavier devices, improve user experience, and optimize device usage through automatic configuration modes, simplifying the process of carrying peripherals.
Smart Images

Figure CN119053933A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This is the first patent application related to this topic. Technical Field
[0003] The present application generally relates to a protective housing or container for extended reality glasses, and more particularly to a foldable and expandable protective housing or container to enhance the capabilities of lightweight extended reality glasses. Background Art
[0004] Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) are collectively referred to as Extended Reality (XR). These technologies have become increasingly popular in recent years. XR technology is being used in everyday gadgets and devices, such as smartphones and smart glasses. Users of smart glasses or XR glasses can use the glasses for leisure or production purposes, such as watching videos, capturing images, or editing text. Many XR glasses are sold with a protective case to protect the glasses when not in use. In addition, when the XR glasses are placed in some protective cases, the protective case can charge the batteries of the XR glasses.
[0005] Some of the XR glasses recently released on the market have a small and lightweight form factor, in stark contrast to their bulky predecessors. Vuzix TM (100g) and MAD Gaze TM (98g) and other XR glasses are based on the weight of Oculus TM Nearly one-fifth the weight of an XR device (503g). Lighter and more compact XR glasses are easier to carry and use. However, the reduction in size and weight is usually accompanied by performance trade-offs, such as limited power, limited interactive capabilities, and reduced computing power and sensor performance.
[0006] To compensate for these limitations, additional peripherals can be used with lightweight XR glasses. Such peripherals can include keyboards, mice, and other input or output devices. Although peripherals can enhance the interactive capabilities of XR glasses, these peripherals are often cumbersome to carry. For example, it can be cumbersome to always carry a keyboard in addition to a pair of XR glasses and their protective case.
[0007] Therefore, there is a need for a protective housing or container for XR glasses that overcomes one or more limitations of the prior art approaches described above. Summary of the invention
[0008] The present invention describes an apparatus and system for providing a foldable housing or container for enhancing the capabilities of a wearable XR display. Exemplary embodiments described herein provide a housing having one or more additional user input modes for XR interaction using a wearable XR display. Some embodiments also provide a housing with additional computing resources to supplement the computing resources of the wearable XR display. The increased capabilities can enable a lightweight wearable XR display to provide the full range of inputs and computing capabilities of a larger, heavier wearable XR display.
[0009] In the present invention, the term "XR" refers to "extended reality". For the purposes of this invention, XR includes virtual reality (VR), augmented reality (AR), and mixed reality (MR). As used herein, XR refers to the presentation of virtual visual elements to a user through an XR display.
[0010] In the present invention, the term "display device" refers to an electronic display for displaying visual information, such as an LED, LCD or CRT monitor.
[0011] In the present invention, the term "wearable XR display" refers to a wearable electronic device for presenting virtual visual elements to one or both eyes of a user. A wearable XR display is typically an immersive stereoscopic display capable of simulating binocular visual effects in a 3D environment, such as a binocular head-mounted display (HMD) in the form of smart glasses or goggles. However, in some examples, the wearable XR display may be a monoscopic display, such as a traditional 2D display capable of rendering a 3D environment by projecting the 3D environment onto two dimensions. A wearable XR display may be limited to one or more display devices, or may also include other components, such as hardware components and / or software components for computing, rendering, and / or displaying information on a display.
[0012] In the present invention, the statement that a first element "substantially covers" a second element means that the first element covers or surrounds at least a majority of any exposed portion of the second element.
[0013] In the present invention, the statement that a second element is "based on" a first element means that the characteristics of the second element are at least partially affected or determined by the characteristics of the first element. The first element can be considered as an input to an operation or calculation, or an input to a series of operations or calculations, which produces a second element that is an output that is not independent of the first element.
[0014] In some aspects, the present invention describes a housing or container for use with a wearable extended reality (XR) display. The housing includes a housing portion forming a cavity for accommodating the wearable XR display, a processor, a communication link, and at least one input device associated with the housing portion and operable to receive user input. The processor is used to generate housing output data based on the user input. The communication link is used to transmit the housing output data to an XR computing device.
[0015] By providing input devices for communicating with the XR computing device, the housing or container can enable a lightweight wearable XR display to provide the same extended input range and computing power as a larger, heavier wearable XR display.
[0016] In some examples, the XR computing device is the wearable XR display.
[0017] In some examples, the XR computing device is a device that communicates with the wearable XR display.
[0018] By providing an XR computing device separate from a wearable XR display, some embodiments may extend the computing power of a lightweight wearable XR display to match the computing power of a larger, heavier wearable XR display.
[0019] In some examples, the housing portion includes a charging unit for charging a power storage component of the wearable XR display.
[0020] In some examples, the shell portion also includes a facial capture camera for capturing an image showing a user's face, and the shell output data is generated based on the image received from the facial capture camera.
[0021] By providing a facial capture camera, some embodiments may enable a wearable XR display to provide an image of a user's face for use by video conferencing applications or other applications that require viewing the user's face.
[0022] In some examples, the at least one input device includes a tactile input surface for receiving tactile input, and the processor is used to generate the housing output data based on the tactile input.
[0023] In some examples, the tactile input surface includes a mechanical sensor.
[0024] In some examples, the tactile input surface includes a touch-sensitive surface.
[0025] In some examples, the container includes a tactile input camera. The tactile input surface includes a surface of the container visible in an image captured by the tactile input camera. The housing output data is generated by: processing the image captured by the tactile input camera to generate tactile data; and processing the tactile data to generate the housing output data.
[0026] In some examples, the housing further comprises an XR display presence sensor configured to generate XR display presence sensor data indicating whether the wearable XR display is present in the cavity. The processor is configured to, in response to receiving the XR display presence sensor data indicating that the wearable XR display is present in the cavity, deactivate the at least one input device such that the housing output data is not generated based on the user input.
[0027] By providing an XR display presence sensor, some embodiments may enable the container to automatically reconfigure the operating mode of the XR system based on whether the user is currently using the wearable XR display.
[0028] In some examples, the container further comprises a cover portion connected to the housing portion and movable between a closed position, in which the cover portion substantially covers the cavity, and an open position, in which the wearable XR display can be placed into or removed from the cavity. The at least one input device comprises a text input surface on the cover portion, and when the cover portion is in the open position, the text input surface can be used to receive text input from movement of a user's finger relative to the text input surface.
[0029] By providing a text input device on the cover portion, some embodiments can take advantage of the form factor of an open container to provide an extended input surface for entering text, such as a keyboard or virtual keyboard.
[0030] In some examples, the text input surface includes a physical keyboard located on the cover portion.
[0031] In some examples, the container further includes a virtual keyboard camera, the virtual keyboard camera being located on the housing portion or the cover portion and configured to capture an image showing a position of a user's finger relative to the text input surface, thereby forming a virtual keyboard surface of the text input surface. The shell output data is generated by: processing the image captured by the virtual keyboard camera to generate text data; and processing the text data to generate the shell output data.
[0032] In some examples, the cover portion is pivotably connected to the housing portion proximal to the cover portion.Moving the cover portion between the closed position and the open position includes pivoting the cover portion away from the housing portion.
[0033] In some examples, the housing further comprises a cover position sensor configured to generate cover position sensor data indicating whether the cover is in the closed position. The processor is configured to, in response to receiving the cover sensor position data indicating that the cover is in the closed position, deactivate the text input device such that the housing output data is not generated based on the text input.
[0034] By providing a cover position sensor, some embodiments may enable the container to automatically reconfigure the operating mode of the XR system based on whether the container is open and therefore in a position to accept text input on the text input surface of the cover portion.
[0035] In some examples, the cover portion further includes an extendable portion forming an extended input surface, the extendable portion being movable between a retracted position and an extended position.
[0036] By providing an extendable portion, some embodiments may further extend the surface area of the container that may be used as a user input surface.
[0037] In some examples, the extendable portion is pivotally connected to the cover portion proximal to the extendable portion, and moving the extendable portion between the retracted position and the extended position includes pivoting the extendable portion away from the cover portion.
[0038] In some examples, when the extendable portion is in the extended position, the text input surface includes the extended input surface.
[0039] In some examples, the extended input surface includes a tactile input surface for receiving tactile input when the extendable portion is in the extended position. The processor is further configured to generate the housing output data based on the tactile input.
[0040] In some aspects, the present disclosure describes a system comprising a housing or container as described above and the wearable XR display.
[0041] In some examples, the system also includes the XR computing device. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Reference will now be made by way of example to the accompanying drawings which show exemplary embodiments of the present application, in which:
[0043] Figure 1 is a top right perspective view of a housing of a wearable XR display according to examples described herein, with a cover portion of the housing in an open position;
[0044] Figure 2A is a block diagram illustrating a system including a wearable XR display, an XR computing device, and a housing according to examples described herein;
[0045] Figure 2B is a block diagram illustrating a system including a wearable XR display and a housing including an XR computing device according to examples described herein;
[0046] Figure 2C is a block diagram illustrating a system including a wearable XR display and a housing according to examples described herein, wherein the wearable XR display includes an XR computing device;
[0047] Figure 3 It is shown FIG. 2A to FIG. 2C A block diagram of components of an exemplary XR computing device;
[0048] Figure 4 It is shown Figure 3 The modules are respectively FIG. 2A to FIG. 2C Shell input devices and Figure 3 Block diagram of the interaction between peripheral devices;
[0049] Figure 5A is an upper right perspective view of a housing of a wearable XR display according to an example described herein, the cover portion of the housing having two pivotally movable extendable portions;
[0050] Figure 5B is an upper right perspective view of a housing of a wearable XR display according to an example described herein, the cover portion of the housing having two slidably movable extendable portions;
[0051] Figure 6 is a top right perspective view of a housing of a wearable XR display having a facial capture camera, a virtual keyboard camera, and a tactile input camera according to examples described herein;
[0052] Figure 7 is an upper right perspective view of a housing of a wearable XR display according to examples described herein, with a cover portion of the housing in a closed position;
[0053] Figure 8 is a top right perspective view of a housing of a wearable XR display according to examples described herein, with a cover portion of the housing in a closed position and a gamepad configuration of the housing located on an exterior face of the cover portion;
[0054] Fig. 9 is an upper right perspective view of a housing of a wearable XR display with a virtual keyboard camera according to examples described herein, also showing virtual visual elements displayed on an XR display device of the wearable XR display.
[0055] Similar reference numerals may be used in different drawings to represent similar components. DETAILED DESCRIPTION
[0056] The present invention describes an apparatus and system for providing a foldable container for enhancing the capabilities of a wearable XR display. In the exemplary embodiments herein, the container is described as a glasses housing. However, it is understood that some embodiments may include other types of containers for housing XR glasses.
[0057] For simplicity, the present invention describes examples in the context of a wearable XR display (e.g., a head mounted display (HMD) unit that enables virtual visual elements to be presented stereoscopically to the user's eyes), wherein the HMD unit includes an inertial measurement unit (IMU) for tracking the movement of the user's head. However, it should be understood that the present invention is not limited to these embodiments. For example, in some embodiments, the wearable XR display simply acts as a traditional 2D display that presents a computer desktop environment to the user's eyes.
[0058] Figure 1 A container for XR glasses is shown, which is embodied as a housing 100 for a wearable XR display 130. The housing 100 includes a shell portion 102 and a cover portion 104. The cover portion 104 is shown in an open position so that the cavity 108 of the shell portion 102 is not covered, so that the wearable XR display 130 can be placed in or removed from the cavity 108. The shape of the cavity 108 can be designed to securely accommodate the wearable XR display 130 to prevent damage when handling the housing 100. For example, the housing 100 can be any suitable shape, including but not limited to a cylinder, a cuboid, a cube, a triangular prism, a pentagonal prism, etc. In some embodiments, the housing 100 may include a wired and / or wireless power receiver (not shown) for receiving power from an external power source. When the wearable XR display 130 is contained in the cavity 108, the received power can be used to charge a power storage component (e.g., a battery (not shown)) of the wearable XR display 130. In some embodiments, the housing 100 may also include a battery or other power storage device (not shown) of the housing 100 itself that can be charged by the received power; when the housing 100 is not connected to an external power source, the battery of the housing can be used to charge the wearable XR display 130.
[0059] The cover portion 104 may be Figure 1 The open position shown and described below Figure 7 In some embodiments, the cover portion 104 is pivotally connected to the housing portion 102 along a proximal side 112 of the cover portion 104, and the cover portion 104 is opened by pivoting away from the housing portion 102. In other embodiments, the cover portion 104 can be moved by other means such as sliding, or can be completely removed from the housing portion 102. In the closed position, the cover portion substantially covers the opening of the cavity 108, so that the wearable XR display 130 is safely enclosed and can be protected from harmful external debris, moisture, etc.
[0060] Some embodiments may omit the cover portion 104 and instead use other methods to secure and / or protect the wearable XR display 130 within the cavity 108 .
[0061] In Figure 1 The cover portion 104 shown in the open position includes a text input surface 110. When the cover portion 104 is in the open position, the text input surface 110 can be used to receive text input from the movement of a user's finger relative to the text input surface 110. In some embodiments, the text input surface 110 includes a physical keyboard for receiving text input from the user's finger. In other embodiments, the text input surface 110 can function as a virtual keyboard, as described below with reference to Figure 6 described.
[0062] In some embodiments, the text input surface 110 can be omitted and the housing 100 can also include one or more other user input devices, such as one or more housing input devices 170 described below, which are directly or indirectly associated with the shell portion 102 through another component (e.g., the cover body portion 104).
[0063] Figure 1 The illustrated housing 100 shows an extendable portion 106A and 106B of the cover portion 104 extending from the left and right sides of the cover portion 104, respectively. Each extendable portion 106A, 106B includes an extended input surface 116. In some embodiments, each extended input surface 116 can function as part of a text input surface 110, for example, as part of a physical keyboard or a virtual keyboard. In some embodiments, each extended input surface 116 can function as a tactile input surface, as described below. Figure 5A and Figure 5B Additional examples of extendable portions 106A, 106B are shown.
[0064] In some embodiments, the housing 100 may include at least one camera 120, which may function as a facial capture camera, a virtual keyboard camera, and / or a tactile input camera, as described below with reference to Figure 6 discussed.
[0065] In some embodiments, the housing may include one or more tactile input surfaces on the housing portion 102, the cover portion 104, and / or the extendable portions 106A, 106B. In some embodiments, the tactile input surface may include a touch-sensitive surface such as a touch screen or a touchpad. In some embodiments, the tactile input surface includes at least one surface visible in an image captured by a tactile input camera, as described below with reference to Figure 6 In some embodiments, the tactile input surface can include at least one mechanical sensor, such as button 118 shown on housing portion 102 .
[0066] In some embodiments, the housing 100 includes one or more ports, such as a USB port or an Ethernet port, for interacting with additional peripheral devices and / or communicating over a network. In some embodiments, the housing 100 includes one or more additional output devices (e.g., a set of speakers or a 2D display) and / or one or more additional input devices or sensors (e.g., an inertial measurement unit (IMU) for determining the orientation and movement of the housing 100).
[0067] The housing 100 enhances the computing and input capabilities of the wearable XR display 130 . Figure 2A , Figure 2B and Figure 2C Three optional configurations of the system 200 (shown as optional configurations 200A, 200B, and 200C, respectively) are shown for enabling communications and computations for an XR user session of a user of the wearable XR display 130 interacting with the housing 100 .
[0068] Figure 2ACommunications between components of a first exemplary system 200A including a wearable XR display 130, an XR computing device 300, and a housing 100 are shown. The housing 100 includes an input processor 140, which can be implemented as one or more processors or controllers, such as a central processing unit, a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, or a combination thereof. The input processor 140 receives input data from a set of housing input devices 170, processes the input data to generate housing output data 160, and transmits the housing output data to the XR computing device 300 via a housing communication link 142, which can be a wired or wireless communication link, such as a network interface. In some embodiments, the housing communication link 142 communicates with the XR computing device 300 using a wireless antenna and a short-range wireless communication protocol.
[0069] The shell input device 170 may include one or more of the following: a physical keyboard 144, a virtual keyboard camera 146, a tactile input camera 148, a facial capture camera 150, one or more touch-sensitive surfaces 152, one or more mechanical sensors 153 (e.g., button 118), one or more cover sensors 154, and / or an XR display presence sensor 156. In some examples, the shell input device 170 may include other input devices, such as a shell IMU. In some examples, the shell input device 170 may include one or more external input devices (e.g., a full-size keyboard) connected to the shell 100 via a port such as a USB port. Figure 6 A virtual keyboard camera 146, a tactile input camera 148, and a facial capture camera 150 are depicted.
[0070] The cover sensor 154 detects movement of the cover portion 104 between the closed position and the open position. In some embodiments, the cover sensor 104 may include a hidden magnet in the surface of the cover portion 104 or the surface of the housing portion 102, which activates a switch when the surface approaches a corresponding surface (i.e., approaches the surface of the housing portion 102 or the surface of the cover portion 104, respectively). Alternatively, a mechanical pin switch or other mechanical sensor may be used at the point where the surface of the cover portion 104 contacts the surface of the housing portion 102 or at the hinge approach side 112 of the cover portion 104. It should be understood that many sensor types are suitable for the purpose of detecting movement of the cover portion 104 between the closed position and the open position.
[0071] The XR display presence sensor 156 is used to detect whether there is a wearable XR display 130 in the cavity 108 of the housing portion 102. In some embodiments, the XR display presence sensor 156 may include one or more hidden magnets. The hidden magnet is located in the cavity, and when the wearable XR display 130 is placed in the cavity 108, the hidden magnet is in close proximity to a complementary magnet on the surface of the wearable XR display 130. When the complementary magnets are in close proximity, the magnets close the circuit to indicate that the wearable XR display 130 is present in the cavity 108. Some embodiments may use a mechanical pin switch. In some embodiments, the mechanical pin switch has complementary (i.e., male and female) ends: the cavity may include one or more first type of mechanical pin switches, which are in a position to contact a complementary type of mechanical pin switch on the surface of the wearable XR display 130. When the complementary mechanical pin switches meet, the circuit is closed, indicating that the wearable XR display 130 is in the cavity 108. It should be understood that other sensor types are suitable for use as the XR display presence sensor 156.
[0072] In some embodiments, one or more of the tactile input surface and / or text input surface 110 of housing 100 can be used to detect whether these surfaces are in a position to interact with the user. A combination of touch-sensitive surface 152, cover sensor 154, mechanical sensor 153, camera, and / or other sensors can be used to make the determination. In some examples, cover portion 104 is substantially two-dimensional and has a top surface (including text input surface 110 shown in FIG. 10) and a bottom surface. In some embodiments, housing portion 102 has a bottom surface, a rear surface, a left surface (including a Figure 1The cover portion 104 and / or the housing portion 102 may include a top surface, a bottom surface, and a bottom surface. One or more of these surfaces of the cover portion 104 and / or the housing portion 102 may be monitored by a sensor for detecting whether a given surface is in a position to interact with a user. In some examples, if a surface is not in contact with another object (e.g., a desktop), the surface may be in a position to interact with a user. For example, one of the surfaces of the housing 100 may have an embedded touch-sensitive surface 152, such as a touchpad, to detect tactile input. If the housing 100 is placed so that the surface including the touch-sensitive surface 152 faces the ground, such as a desktop, the touch-sensitive surface 152 will output tactile input data indicating that the surface of the housing 100 is not capable of receiving user input. Conversely, if the surface including the touch-sensitive surface 152 is not facing the ground, the touch-sensitive surface 152 will output tactile input data indicating that the surface of the housing 100 is capable of receiving user input. In some examples, sensors may also be used to determine whether the surface of the housing 100 is facing the user wearing the wearable XR display 130, for example, by tracking the position of the wearable XR display 130 relative to the housing 100 and the orientation of the housing 100 using the head-mounted IMU 134 and the IMU of the housing 100, respectively. Figure 4 ), by observing some combination of proximity sensor data (e.g., data received from touch-sensitive surface 152), gravity sensor data (e.g., inertial measurement unit data), camera data, and / or other sensor data.
[0073] Figure 2B A second exemplary system 200B is shown including a wearable XR display 130 and a housing 100, wherein the housing 100 includes an XR computing device 300. In the system 200B, the input processor 140 may be a processor used by the XR computing device 300 for its own operation (as described below with reference to Figure 3 The housing communication link 142 may have the ability to communicate directly with the wearable XR display 130 via a wired or wireless communication link, but may also include a communication bus or other internal communication system for transmitting data between the various components of the housing 100 (including the components of the XR computing device 300), as described below with reference to Figure 3 described.
[0074] Figure 2CA third exemplary system 200C is shown that includes a wearable XR display 130 and a housing 100, wherein the wearable XR display 130 includes an XR computing device 300. In this system 200C, some or all of the computations required to implement an XR user session and manage interactions between components of the housing 100 and the wearable XR display 130 are performed by the XR computing device 300 residing within the wearable XR display 130.
[0075] Figure 3 It is shown FIG. 2A to FIG. 2C 300. Although one exemplary embodiment of an XR computing device 300 is shown and discussed below, other embodiments may be used to implement the examples disclosed herein, which may include components different than those shown. Figure 3 A single instance of each component of the XR computing device 300 is shown, and each component shown may have multiple instances. In addition, although the XR computing device 300 is described as a single device, according to the above FIG. 2A to FIG. 2C It will be appreciated that the various functions and components of the XR computing device 300 may be replicated or divided into two, three or more different devices, including the housing 100, the wearable XR display 130 and / or a separate XR computing device 300 that communicates with both the housing 100 and the wearable XR display 130.
[0076] The XR computing device 300 includes one or more processors, such as a central processing unit, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, or a combination thereof. The one or more processors may be collectively referred to as a "processor device" or simply as a processor 302. In some embodiments, such as in the above-mentioned system 200B, the XR computing device 300 forms a part of the housing 100, and the processor 302 may also perform the functions of the input processor 140.
[0077] The XR computing device 300 also includes one or more input / output (I / O) interfaces, collectively referred to as I / O interfaces 304, which are connected to an input device 324 (such as a mouse or keyboard), an output device 322 (such as a 2D display or a speaker), a head-mounted inertial measurement unit (IMU) 132 of the wearable XR display 130, an XR display device 134 of the wearable XR display 130, and a shell input device 170 (via a shell communication link 142).
[0078] The XR computing device 300 may include one or more optional network interfaces 306 for wired or wireless communication with a network (e.g., an intranet, the Internet, a P2P network, a WAN, and / or a LAN) or other nodes. The one or more network interfaces 306 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for communication within and / or between networks. In some embodiments, the one or more network interfaces 306 may be used as or in place of the I / O interface 304 to communicate with one or more of the input devices and / or output devices described above, for example using 802.11 or Bluetooth. TM Wireless communication.
[0079] The XR computing device 300 includes one or more memories, collectively referred to as memory 308, which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The non-transitory memory 308 may store machine-executable instructions for execution by the processor 302, such as for performing the examples described herein. A set of machine-executable instructions defining a sensing module 310, a display module 312, a communication module 314, and an input module 316 stored in the memory 308 is shown, which may be executed by the processor 302 to perform the computing operations described herein. Figure 4 The operation of the XR computing device 300 and the overall system 200 is described when executing modules 310, 312, 314, 316. The memory 308 may include other software instructions, such as for implementing an operating system and other applications or functions.
[0080] In some examples, the XR computing device 300 may also include one or more electronic storage units (not shown), such as a solid-state drive, a hard disk drive, a magnetic disk drive, and / or an optical disk drive. In some examples, one or more data sets and / or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with the XR computing device 300), or may be provided by a transient or non-transient computer-readable medium. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.
[0081] The components of the XR computing device 300 may communicate with each other in various ways, such as the data bus 318. In some embodiments, the operation of the XR computing device 300 may not be performed by a distributed computing system, such as a cloud computing system or a virtual machine instance implemented on one or more servers. In some embodiments, such as in the system 200B described above, the XR computing device 300 forms part of the housing 100, and the data bus 318 may perform some of the functions of the housing communication link 142 to enable communication between the components of the housing 100 (including the components of the XR computing device 300).
[0082] Figure 4 Modules 310, 312, 314, and 316 are shown as Figure 3 A block diagram of the interaction between the housing input device 170 and various peripheral devices is shown.
[0083] The communication module 314 mediates the interaction of the XR computing device 300 with all peripheral devices. Figure 4 The peripheral devices shown include a shell input device 170, an output device 322, an input device 324, a head mounted IMU 132 of a wearable XR display 130, and an XR display device 134 of a wearable XR display 130. The communication module 314 can be used to receive and process various types of input and output data from other modules 312, 314, 316 and peripheral devices routed to and from other modules 312, 314, 316. The input data may include user input data and / or sensor data received from certain peripheral devices, and the output data may include control data and / or output data transmitted to certain peripheral devices. In the shell of the shell input device 170, the shell output data 160 generated by the input processor 140 based on the user input and sensor data of the shell input device 170 is sent to the XR computing device 300 via the shell communication link 142 and processed by the communication module 310 before being routed to the other modules 312, 314, 316. The communication module 314 may generally be responsible for establishing and ending communication sessions and channels with external devices (e.g., the wearable XR display 130), and for sending or routing data to its appropriate destination.
[0084] The sensing module 310 is used to process the housing output data 160, in particular the data generated by the cover sensor 154 and the XR display presence sensor 156, to determine whether the cover portion 104 is in an open position or a closed position, and whether the wearable XR display 130 is present in the cavity 108. In some embodiments, the sensing module 310 can also be used to process a portion of the housing output data 160 that indicates whether one or more surfaces of the housing 100 are in a position to be interacted with by a user, as described above. Based on the data received from the cover sensor 154 and the XR display presence sensor 156, the sensing module 310 can determine that the housing 100 is in one of four states: state 1 indicates [the cover is open and the XR display is present], state 2 indicates [the cover is closed and the XR display is present], state 3 indicates [the cover is open and the XR display is not present], and state 4 indicates [the cover is closed and the XR display is not present]. Based on the state (1, 2, 3, or 4) of the housing 100, the sensing module 310 may transmit control data to the input module 316 and / or the display module 312 that is intended to activate or deactivate certain functions of the housing 100, the wearable XR display 130, and / or the XR computing device 300. In addition, in some embodiments, in response to determining that a given external surface of the housing 100 is in a position to be interacted with by a user, the sensing module 310 may transmit control data to the input module 316 and / or the display module 312 that is intended to activate or deactivate certain functions of the housing 100, the wearable XR display 130, and / or the XR computing device 300 related to the external surface and the input or output surfaces located thereon.
[0085] For example, in state 1 [lid is open and XR display is present]: an auxiliary display (e.g., a 2D display connected to the XR computing device 300 as an output device 322) can be activated by the display module 312, the XR display device 134 of the wearable XR display 130 can be deactivated by the display module 312, the keyboard 144 or a virtual keyboard (implemented by the virtual keyboard camera 146) can be activated by the input module 316, the facial capture camera 150 can be activated by the input module 316, and one or more tactile input surfaces (e.g., an extended input surface 116 implemented as a touch-sensitive surface 152 in this example) can be activated by the input module 316. In some embodiments, the charging function of the housing 100 can also be activated to charge the wearable XR display 130. Thus, detecting state 1 [lid open and XR display present] can trigger the sensing module 310 to place the system 200 in a “laptop” configuration to enable the user to view the 2D display while using the keyboard 144 of the cover portion 104 and various other input devices of the housing 100, and essentially use the housing 100 and XR computing device 300 as a laptop computer when the housing 100 is open and the wearable XR display 130 is charging within the cavity 108.
[0086] For example, in state 2 [cover closed and XR display present]: all input and output devices may be deactivated, and in some embodiments, charging functionality may be activated. Thus, detecting state 2 [cover closed and XR display present] may trigger the sensing module 310 to place the system 200 in a “sleep” configuration to charge the wearable XR display 130 and conserve power.
[0087] For example, in state 3 [lid open and XR display not present]: the auxiliary display (e.g., a 2D display connected to the XR computing device 300 as an output device 322) can be deactivated by the display module 312, the XR display device 134 of the wearable XR display 130 can be activated by the display module 312, the keyboard 144 or virtual keyboard (implemented by the virtual keyboard camera 146) can be activated by the input module 316, the facial capture camera 150 can be activated by the input module 316, and one or more tactile input surfaces (e.g., the extended input surface 116 implemented as the touch-sensitive surface 152 in this example) can be activated by the input module 316. In some embodiments, the charging function of the housing 100 can also be deactivated. Therefore, detecting state 3 [lid open and XR display not present] can trigger the sensing module 310 to place the system 200 in the "XR session with open housing" configuration to enable the user to wear and interact with the wearable XR display 130 while using the keyboard 144 of the cover portion 104 and various other input devices of the housing 100.
[0088] For example, in state 4 [lid closed and XR display not present]: an auxiliary display (e.g., a 2D display connected to the XR computing device 300 as an output device 322) can be deactivated by the display module 312, the XR display device 134 of the wearable XR display 130 can be activated by the display module 312, the keyboard 144 or virtual keyboard (implemented by the virtual keyboard camera 146) can be deactivated by the input module 316, the facial capture camera 150 can be deactivated by the input module 316, and one or more tactile input surfaces (e.g., the extended input surface 116 implemented as the touch-sensitive surface 152 in this example) can be deactivated by the input module 316. In some embodiments, the charging function of the housing 100 can also be deactivated. Thus, detecting state 4 [cover closed and XR display not present] may trigger the sensing module 310 to place the system 200 in an “XR session with closed housing” configuration, so that the user can wear and interact with the wearable XR display 130 without using the keyboard 144 of the cover portion 104 or other input devices of the housing 100. However, it should be noted that reference Figure 7Various embodiments are described in which certain input surfaces located on the outer surface of the housing 100 can be activated in state 4 and / or other states. In addition, even in state 4 (or state 2), in some examples, the sensing module 310 can keep one or more input devices of the housing 100 activated, such as the button 118.
[0089] In some embodiments, the sensing module 310 can issue a command to the communication module 314 to start or end a communication session with one or more external devices when a state change is detected. For example, a communication channel can be opened to the wearable XR display 130 when the housing is opened or when the wearable XR display is removed from the housing 100, and closed when the housing 100 is closed or when the wearable XR display 130 is placed into the cavity 108.
[0090] In some embodiments, the sensing module 310 may also determine the state of the housing 100, or the state of the system 200 as a whole, based in part on the state of additional devices (e.g., various input devices 324 and / or output devices 322). For example, the display module 312 may route visual output data to an external 2D display plugged into a port of the XR computing device 300 only when the sensing module 310 or the display module 312 detects that the external 2D display is turned on and connected to the XR computing device 300. In some examples, such determinations of the state of the system 200 as a whole may also affect the operation of the individual modules 310, 312, 314, 316 in other ways, such as decisions to activate or deactivate various input and output devices.
[0091] The input module 316 is used to receive input data from various input devices and sensors (via the communication module 314) and process the input data to provide software commands to other software modules of the XR computing device 300. For example, the input module 316 can be used to process images captured by the virtual keyboard camera 146 showing the movement of the user's fingers relative to the text input surface 110 using computer vision techniques used to implement a virtual keyboard to generate text data. Similarly, the input module 316 can be used to process images captured by the tactile input camera 148 showing the movement of the user's fingers relative to the tactile input surface of the housing 100 using computer vision techniques used to implement a gesture recognition system and / or a virtual pointer device to generate tactile data such as pointer data or gesture data. The input module 316 can also process other input data to generate other forms of data that can be used by other software modules, such as processing IMU data received from the head-mounted IMU 132 to generate user head orientation and head movement data that can be used by other modules to implement and manage XR user sessions. In some embodiments, the input module 316 may include one or more submodules, each submodule for processing input from a corresponding input mode. In some embodiments, the input module 316 may include a camera input submodule for processing input received from one or more cameras (virtual keyboard camera 146, tactile input camera 148 and / or facial capture camera 150). In some embodiments, the input module 316 may include a keyboard input submodule for processing input received from a physical keyboard 144 and / or a virtual keyboard implemented by a virtual keyboard camera 146. In some embodiments, the input module 316 may include a tactile or tangible input submodule for processing input received from one or more touch-sensitive surfaces 152, one or more mechanical sensors 153 (e.g., button 118) and / or a tactile input surface implemented by a tactile input camera 148. In some embodiments, the input module 316 may include an IMU input submodule for processing input received from a housing IMU and / or a head-mounted IMU 134.
[0092] The display module 312 is used to generate visual output data for display on one or more display devices, such as one or more XR display devices 134 of the wearable XR display 130 and / or a 2D display connected to the XR computing device 300 as an output device 322. The visual output data can be routed to one or more such active display devices based on the state of the housing 100 detected by the sensing module 310. In some embodiments, the display module 312 is a more general output module that generates other forms of output data and / or routes other forms of output data to various other output devices (e.g., the output device 322). For example, the output module can be used to route audio output data to one or more active audio output devices, such as speaker devices of the wearable XR display 130 and / or speakers connected to the XR computing device 300 as output devices 322, based on the state of the housing 100 detected by the sensing module 310.
[0093] Reference FIG. 5A to FIG. 7 Additional details of the various housing input devices 170 are described.
[0094] Figure 5A A housing 100 is shown having a cover portion 104 with two pivotally movable extendable portions 106A, 106B. The left extendable portion 106A is shown fully extended by pivoting away from the cover portion 104 along an arc 502A. The right extendable portion 106B is shown retracted and covering the top of the cover portion 104, but the right extendable portion 106B can be extended by pivoting away from the cover portion 104 along an arc 502B.
[0095] Figure 5B A housing 100 is shown having a cover portion 104 with two slidably movable extendable portions 106A, 106B. The left extendable portion 106A is shown fully extended by sliding away from the cover portion 104 along axis 504A. The right extendable portion 106B is shown retracted within, above or below the cover portion 104, but the right extendable portion 106B can be extended by sliding away from the cover portion 104 along axis 504B.
[0096] It should be understood that in some examples, the sensing module 310 can be used to detect when each extendable portion 106A, 106B is extended or retracted based on various inputs (e.g., from a mechanical sensor or a magnetic sensor similar to the cover sensor 154 described above), and activate and / or deactivate various components accordingly (e.g., when a portion of the text input surface 110 is covered, such as Figure 5AAs shown, the text input surface 110 may be deactivated, and / or the extended input surface 116 of the extended portions 106A, 106B may be deactivated when the extended portions 106A, 106B are retracted).
[0097] Figure 6 The housing 100 is shown having a facial capture camera 150, a virtual keyboard camera 146, and a tactile input camera 148. The facial capture camera 150 has a field of view 602 suitable for capturing an image in which a user's head 610 or face (shown wearing the wearable XR display 130) is visible. Thus, the facial capture camera 150 can enhance the input capabilities of the wearable XR display 130 by providing a view of the user's face for applications such as video chat. The virtual keyboard camera 146 has a field of view 604 suitable for capturing an image in which movement of the fingers of the user's hand 608 relative to the text input surface 110 is visible in the image. The tactile input camera 148 has a field of view 606 suitable for capturing an image in which movement of the fingers of the user's hand 608 relative to the tactile input surface (e.g., the extended input surface 116) is visible in the image. As described above, in some embodiments, two or more of these cameras 150 , 146 , 148 may be implemented as a single camera that captures two or more of the respective fields of view 602 , 604 , 606 .
[0098] In some embodiments, the virtual keyboard camera 146 is used to capture images of the user's fingers in the air or on a surface other than the surface of the housing 100, thereby realizing a virtual keyboard in the air or on a surface other than the surface of the housing 100. Similarly, in some embodiments, the tactile input camera 148 can be used to capture air gestures or the movement of fingers on a surface other than the surface of the housing 100. In some examples, the sensing module 310 can deactivate the physical keyboard 144 during the closed housing state 4 [the cover is closed and the XR display is not present], but can also activate the virtual keyboard camera 146 in state 4 to realize a virtual keyboard on another surface or in the air. Similarly, in state 4, the sensing module 310 can activate the tactile input camera 148 in state 4 to realize a tactile input mode by detecting the movement of a finger or hand on another surface or in the air. In some examples, when the housing is opened (e.g., transitioning from state 4 to state 3), the virtual keyboard and virtual tactile input mode can be deactivated (immediately or after a delay), while activating the physical keyboard 144 and / or other input surfaces of the opened housing 100.
[0099] In some examples, virtual keyboard camera 146 and / or tactile input camera 148 can detect movement of objects other than a user's fingers. For example, in some embodiments, a smartphone in communication with housing 100 and / or XR computing device 300 can be used as a ray projector to select letters from a virtual keyboard. When a virtual ray intersects a letter in the virtual keyboard, as detected by virtual keyboard camera 146, tapping on the smartphone's touch screen can select a letter from the virtual keyboard. Similarly, a smartphone can be used as a ray projector to perform gestures interpreted by tactile input camera 148, such as pointing gestures. In some embodiments, the smartphone can also implement some or all of the functionality of XR computing device 300.
[0100] Figure 7 The housing 100 is shown with the cover portion 104 in a closed position. The cover portion 104 may be held in place in the closed position by fastening means such as magnetic or mechanical fastening means or by other suitable fasteners. In the illustrated embodiment, the cover portion 104 is substantially two-dimensional and has a text input surface 110 (e.g., a text input surface 110) on a top or interior face. Figure 1 ), and having an auxiliary text input surface 110A on the bottom or outer surface opposite to the top or inner surface. When the cover portion 104 is in a closed position, the text input surface 110 can be deactivated (such as in the above-mentioned exemplary states 1 and 3), and the auxiliary text input surface 110A can be activated, so that the auxiliary text input surface 110A can be used to receive text input from the user's finger. In other embodiments, the tactile input surface can replace the auxiliary text input surface 110A or be located on the outer surface of the cover portion 104 together with the auxiliary text input surface 110A. In some embodiments, as described above, the auxiliary text input surface 110A and / or the tactile input surface on the outer surface of the cover portion 104 can be activated only when the sensing module 310 determines that the outer surface of the cover portion 104 is not in contact with another object such as a desktop. In some embodiments, one or more other external faces or surfaces of the housing 100, such as one or more external faces of the shell portion 102, may include an auxiliary text input surface 110A and / or a tactile input surface, which is activated when the sensing module 310 determines that the external face or surface is not in contact with another object such as a desktop.
[0101] In some embodiments, other components such as a 2D display device may be located on an external surface or an internal surface of the housing 100. For example, a 2D display may be located Figure 7 , which is the position of the auxiliary text input surface 110A. Such a 2D display may be activated in some states, such as when the wearable XR display 130 is not used as a display device.
[0102] Figure 8 The housing 100 is shown with the cover portion 104 of the housing 100 in a closed position and a gamepad configuration of the housing 100 located on an exterior face of the cover portion. In the illustrated embodiment, the cover portion 104 includes a tactile input surface 802 (e.g., a touch-sensitive surface 152 such as a touchpad) and a set of gamepad buttons 804 (e.g., mechanical sensors 153) on the exterior face. When the cover portion 104 is in the closed position (e.g., in state 3), the tactile input surface 802 and the gamepad buttons 804 can be used as a game controller by a user wearing the wearable XR display 130. In some embodiments, the IMU of the housing 100 can also be used to detect movement and orientation of the housing 100 when it is held by the user's hand to provide further game control input.
[0103] Fig. 9 The housing 100 is shown with a virtual keyboard camera 146, and virtual visual elements displayed on the XR display device 134 of the wearable XR display 130 are also shown. In some examples, the display module 312 can be used to display all or part of the image data captured by one or more of the cameras 146, 148, 150 on the XR display device 134 of the wearable XR display 130. In some embodiments, the images captured by one or more cameras can be processed to generate an abstract or stylized representation of the information captured in the image, which can be displayed by the XR display device 134. For example, in Fig. 9 In the example shown, the XR environment displayed by the XR display device 134 includes a skeletal wireframe visual representation 902 of the user's hands 608A, 608B relative to the text input surface 110. The skeletal wireframe visual representation 902 may be generated by applying computer vision techniques to identify the positions of the user's hands relative to the text input surface 110 captured in an image by the virtual keyboard camera 146, and then generating an abstract or stylized visual representation of a keyboard with an overlaid skeletal wireframe representation of the positions of the user's hands (including fingers and their joints) relative to the text input surface 110. By presenting a clearly visible representation of the user's hands and fingers relative to a physical or virtual keyboard within the user's field of view (regardless of the user's gaze direction), text input may be accelerated during an XR user session.
[0104] Overview
[0105] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or changed as appropriate. Where appropriate, one or more steps may be performed in an order other than the order described.
[0106] Although the present invention is at least partially described in terms of methods, it will be understood by those of ordinary skill in the art that the present invention is also directed to various components for performing at least some aspects and features of the methods, whether by hardware components, software, or any combination thereof. Accordingly, the technical solution of the present invention may be embodied in the form of a software product. Suitable software products may be stored in a pre-recorded storage device or other similar non-volatile or non-transient computer-readable medium, including a DVD, CD-ROM, USB flash drive, removable hard disk, or other storage medium, etc. The software product includes instructions tangibly stored thereon, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform an example of the method disclosed herein.
[0107] Without departing from the subject matter of the claims, the present invention may be implemented in other specific forms. The exemplary embodiments described are merely illustrative and not restrictive in all respects. Features selected from one or more of the above-described embodiments may be combined to create optional embodiments that are not explicitly described, and features suitable for such combinations may be understood within the scope of the present invention.
[0108] All values and subranges within the disclosed range are also disclosed. In addition, although the systems, devices and processes disclosed and shown herein may include a specific number of elements / assemblies, the systems, devices and assemblies may be modified to include more or less such elements / assemblies. For example, although any disclosed element / assembly may be cited as a single quantity, the embodiments disclosed herein may be modified to include a plurality of such elements / assemblies. The subject matter described herein is intended to cover and encompass all appropriate technical changes.
Claims
1. A container for use with a wearable extended reality (XR) display, characterized in that: include: a housing portion forming a cavity for accommodating the wearable XR display; at least one input device associated with the housing portion and operable to receive user input; a processor for generating shell output data based on the user input; A communication link is provided for transmitting the shell output data to an XR computing device.
2. The container according to claim 1, characterized in that The XR computing device is the wearable XR display.
3. The container according to claim 1, characterized in that The XR computing device is a device that communicates with the wearable XR display.
4. The container according to any one of claims 1 to 3, characterized in that The housing portion includes a charging unit for charging a power storage component of the wearable XR display.
5. The container according to any one of claims 1 to 4, characterized in that The housing portion also includes a facial capture camera for capturing an image showing a user's face; The shell output data is generated based on the image received from the facial capture camera.
6. The container according to any one of claims 1 to 5, characterized in that The at least one input device comprises a tactile input surface for receiving tactile input, wherein, The processor is configured to generate the housing output data based on the tactile input.
7. The container according to claim 6, characterized in that The tactile input surface includes a mechanical sensor.
8. The container according to claim 6, characterized in that The tactile input surface comprises a touch-sensitive surface.
9. The container according to claim 6, characterized in that The container includes a tactile input camera; The tactile input surface comprises a surface of the container visible in an image captured by the tactile input camera; The shell output data is generated in the following way: processing the image captured by the tactile input camera to generate tactile data; The haptic data is processed to generate the housing output data.
10. The container according to any one of claims 1 to 9, characterized in that The container further includes an XR display presence sensor for generating XR display presence sensor data indicating whether the wearable XR display is present within the cavity; The processor is used to: In response to receiving XR display presence sensor data indicating that the wearable XR display is present within the cavity, deactivating the at least one input device such that the housing output data is not generated based on the user input.
11. The container according to any one of claims 1 to 10, characterized in that The container further comprises a cover portion connected to the housing portion and movable between the following positions: a closed position in which the cover portion substantially covers the cavity; an open position, in which the wearable XR display can be placed into or removed from the cavity, wherein The at least one input device includes a text input surface on the cover portion, which is operable to receive text input from movement of a user's finger relative to the text input surface when the cover portion is in the open position.
12. The container according to claim 11, characterized in that The text input surface includes a physical keyboard located on the cover portion.
13. The container according to claim 11, characterized in that The container further comprises a virtual keyboard camera, the virtual keyboard camera being located on the housing portion or the cover portion and configured to capture an image showing a position of a user's finger relative to the text input surface, thereby forming a virtual keyboard surface of the text input surface; The shell output data is generated in the following way: Processing the image captured by the virtual keyboard camera to generate text data; The text data is processed to generate the shell output data.
14. The container according to any one of claims 11 to 13, characterized in that The cover portion is pivotally connected to the housing portion proximally of the cover portion; Moving the cover portion between the closed position and the open position includes pivoting the cover portion away from the housing portion.
15. The container according to any one of claims 11 to 14, characterized in that The container further comprises a cover position sensor for generating cover position sensor data indicating whether the cover is in the closed position; The processor is used to: In response to receiving cover sensor position data indicating that the cover is in the closed position, the text input device is deactivated such that the housing output data is not generated based on the text input.
16. The container according to any one of claims 11 to 15, characterized in that The cover portion further includes an extendable portion forming an extended input surface; The extendable portion is movable between a retracted position and an extended position.
17. The container according to claim 16, characterized in that The extendable portion is pivotably connected to the cover portion proximal to the extendable portion; Moving the extendable portion between the retracted position and the extended position includes pivoting the extendable portion away from the cover portion.
18. The container according to claim 16 or 17, characterized in that When the extendable portion is in the extended position, the text input surface comprises the extended input surface.
19. The container according to any one of claims 16 to 18, characterized in that The extended input surface comprises a tactile input surface for receiving tactile input when the extendable portion is in the extended position; The processor is further configured to generate the housing output data based on the tactile input.
20. A system, characterized in that: include: A container according to any one of claims 1 to 19; The wearable XR display.
21. The system according to claim 20, characterized in that Also included is the XR computing device.
