Partial perspective in virtual reality
By detecting hand perspective trigger events in a virtual reality system, and automatically overlaying part of the video feed onto the virtual environment using cameras and sensors, the difficulty for users to view the real world in a virtual reality system is solved, achieving seamless interaction between virtual reality and the real world.
Patent Information
- Application Number
- CN202180094367.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-10-11
- Filing Date
- 2021-12-21
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2041-12-21
AI Technical Summary
Existing virtual reality systems typically require users to remove the device or navigate through cumbersome menus when they need to view the real world, leading to user experience interruptions and potential risks of real-world collisions.
By detecting hand perspective trigger events, the system uses the camera and sensors of the virtual reality device to identify the user's hand gestures and positions, and automatically overlays part of the video feed onto the virtual environment, allowing users to view the real world without removing the device.
It enables users to interact with the real world in a virtual reality environment in real time, avoiding device removal and menu operations, thus improving the continuity and security of the user experience.
Smart Images

Figure CN116940915B_ABST
Abstract
Description
[0001] Cross-reference to related applications
[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 129,486 (Attorney General No. 3589-0031PV01), filed on December 22, 2020, entitled “Partial Passthrough in Virtual Reality,” which is incorporated herein by reference in its entirety. Technical Field
[0003] This disclosure relates to partial video passthrough in a virtual reality (VR) environment. Background Technology
[0004] Virtual reality (VR) systems offer users the ability to experience different worlds, learn in new ways, and connect better with others. These systems track user movement and translate it into interactions with "virtual objects" (i.e., computer-generated representations of objects appearing in the virtual environment). For example, a VR system can track a user's hand movements, translating a grasping gesture into picking up a virtual object. When a user sees and interacts with a virtual object, they physically move in the real world. In some situations, a VR system may obstruct a user's view of part or all of the real world, or the user may become distracted by virtual objects, leading to unintentional collisions with real-world objects. In other situations, users may need to observe what is happening in the real world. For example, a user may need to interact with other people in a room or with objects. However, this may involve several steps, including turning off the VR device or removing it. Summary of the Invention
[0005] Therefore, the present invention relates to a method and computing system for providing partially transparent video to a user of a virtual reality device, as well as a computer-readable storage medium and a computer program.
[0006] In one aspect, the present invention relates to a method for providing a partially transparent video to a user of a virtual reality device, the method comprising:
[0007] Detect hand perspective trigger events;
[0008] In response to the detection of a hand perspective trigger event, identify hand perspective video feeds from one or more cameras of the virtual reality device;
[0009] Align the hand-view video feed with the virtual environment presented to the user via a virtual reality device; and
[0010] Based on the alignment of the hand perspective video feed with the virtual environment, the hand perspective video feed is overlaid on the virtual environment.
[0011] In an embodiment of the method according to the invention, detecting a hand perspective trigger event may include: recognizing a specific gesture made by the user based on the monitored hand posture of the user, wherein the user's hand posture can be monitored by A) using input from one or more sensors attached to the user's hand or wrist, and / or B) by analyzing images acquired by a virtual reality device.
[0012] In an embodiment of the method according to the present invention, detecting a hand perspective trigger event may include: recognizing that the user's hand leaves a designated area.
[0013] In an embodiment of the method according to the present invention, detecting a hand perspective trigger event may include: determining that the user's hand is within a threshold distance of a real-world surface.
[0014] In an embodiment of the method according to the present invention, detecting a hand perspective trigger event may include: determining that the user's hand is within a threshold distance of a real-world object.
[0015] In an embodiment of the method according to the invention, detecting a hand perspective trigger event may include: identifying a predicted collision between a part of the user and a real-world object by analyzing the user's kinematic model using the determined current hand speed and current hand trajectory.
[0016] In an embodiment of the method according to the invention, detecting a hand perspective trigger event may include: determining that the user speaks a specific voice command or presses a specific button associated with a virtual reality device.
[0017] In embodiments of the method according to the present invention, detecting hand perspective triggers can be based on the user's hand movements, and identifying hand perspective video feeds can include:
[0018] Identify the user's hand performing the action in one or more images from one or more cameras of a virtual reality device; and
[0019] Identify the area around the identified hand in one or more images from one or more cameras of a virtual reality device.
[0020] In an embodiment of the method according to the present invention, identifying the hand perspective video feed may include:
[0021] Determine the point where a specific straight line or curve extending from the user's hand intersects a real-world object or surface; and
[0022] Identify the area around a determined point in one or more images from one or more cameras of a virtual reality device.
[0023] In an embodiment of the method according to the invention, detecting a hand perspective trigger event may be based on the identification of an object in one or more images from one or more cameras of a virtual reality device; and identifying the hand perspective video feed may include: identifying the area around the identified object in one or more images from one or more cameras of a virtual reality device.
[0024] In embodiments of the method according to the invention, detecting a hand perspective trigger event can be based on the identification of an object or surface in one or more images from one or more cameras of a virtual reality device; and
[0025] The size of the hand perspective video feed can be set based on a determined distance between the user's hand and the identified object or surface.
[0026] In an embodiment of the method according to the present invention, identifying the hand perspective video feed may include:
[0027] Identify the user's first hand and second hand in one or more images from one or more cameras of a virtual reality device; and
[0028] Identify the area surrounding the identified first hand in one or more images from one or more cameras of a virtual reality device, wherein the size of the area is based on a determined distance between the first hand and the second hand.
[0029] In an embodiment of the method according to the invention, aligning the hand perspective video feed with a virtual environment may include: matching coordinates associated with the hand perspective video feed with coordinates in the virtual environment based on matching one or more identified objects and / or body parts between the hand perspective video feed and the virtual environment.
[0030] In one aspect, the invention also relates to a computing system for providing partially transparent video to a user of a virtual reality device, the computing system comprising:
[0031] One or more processors; and
[0032] One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform the method as described above or to perform a process including the following steps:
[0033] Detect hand perspective trigger events;
[0034] In response to the detection of a hand perspective trigger event, identify hand perspective video feeds from one or more cameras of the virtual reality device;
[0035] Align the hand-view video feed with the virtual environment presented to the user via a virtual reality device; and
[0036] Based on the alignment of the hand perspective video feed with the virtual environment, the hand perspective video feed is overlaid on the virtual environment.
[0037] In an embodiment of the computing system according to the present invention, detecting a hand perspective trigger event may include: determining that the user's hand is within a threshold distance of a real-world object or a real-world surface.
[0038] In an embodiment of the computing system according to the invention, detecting a hand perspective trigger event may include: recognizing a specific gesture made by the user based on the monitored hand posture of the user, wherein the user's hand posture may be A) monitored using input from one or more sensors attached to the user's hand, wrist, or forearm, and / or B) monitored by analyzing images acquired by a virtual reality device.
[0039] In an embodiment of the computing system according to the present invention, detecting a hand perspective trigger event can be based on the user's hand movements; and recognizing the hand perspective video feed can include:
[0040] Identify the user's hand performing the action in one or more images from one or more cameras of a virtual reality device; and
[0041] Identify the area around the identified hand in one or more images from one or more cameras of a virtual reality device.
[0042] In an embodiment of the computing system according to the present invention, detecting a hand perspective trigger event may include: recognizing that the user's hand leaves a designated area.
[0043] In one aspect, the present invention also relates to a computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform the method described above or to perform a process comprising the following steps:
[0044] Detect hand perspective trigger events;
[0045] In response to the detection of a hand perspective trigger event, identify hand perspective video feeds from one or more cameras of the virtual reality device;
[0046] Align the hand-view video feed with the virtual environment presented to the user via a virtual reality device; and
[0047] Based on the alignment of the hand perspective video feed with the virtual environment, the hand perspective video feed is overlaid on the virtual environment.
[0048] In an embodiment of the computer-readable storage medium according to the present invention, identifying the hand perspective video feed may include:
[0049] Determine the point where a specific straight line or curve extending from the user's hand intersects a real-world object or surface; and
[0050] Identify the area around a determined point in one or more images from one or more cameras of a virtual reality device.
[0051] In one aspect, the invention also relates to a computer program comprising instructions that, when executed by a computing system, cause the computing system to perform the method described above. Attached Figure Description
[0052] Figure 1 A block diagram illustrating an overview of devices on which some embodiments of this technology may run.
[0053] Figure 2A A line drawing illustrating a virtual reality headset that can be used in some embodiments of this technology.
[0054] Figure 2B A line diagram illustrating a mixed reality (MR) head-mounted device that can be used in some embodiments of this technology is shown.
[0055] Figure 2C The diagram illustrates a controller that, in some implementations, can be held with one or both hands to interact with an artificial reality environment.
[0056] Figure 3 A block diagram illustrating an overview of the environment in which some embodiments of this technology may run.
[0057] Figure 4 A block diagram is provided to illustrate components that may be used in a system employing the disclosed technology in some embodiments.
[0058] Figure 5 A flowchart illustrating a process for providing partial video perspective to a virtual reality device in some embodiments of this technology is provided.
[0059] Figure 6A conceptual diagram illustrating an example of a halo-like partial video perspective overlaid on a virtual environment in some embodiments employing the disclosed techniques.
[0060] Figure 7 A conceptual diagram illustrating an example of a flashlight-like partial video perspective overlaid on a virtual environment in some embodiments employing the disclosed technology.
[0061] The technology described herein can be better understood by referring to the following specific embodiments in conjunction with the accompanying drawings, wherein similar reference numerals denote the same or functionally similar elements. Detailed Implementation
[0062] Various aspects of this disclosure relate to a partial perspective system that allows users of virtual reality devices (such as virtual reality headsets) to view video of their environment without removing or deactivating the virtual reality device or activating cumbersome controls. There are various situations where users interacting with a virtual reality environment may need to see at least a portion of what is happening in the real world. For example, a user may be about to collide with a real-world object, or may want to interact with real-world objects or people. However, existing virtual reality systems require users to either remove their virtual reality device or activate controls through a cumbersome set of menus to view the real world, unnecessarily detaching the user from the virtual reality experience. Therefore, there is a need for a system that can simply and / or automatically display a partial video feed of the user's environment overlaid on a portion of the virtual environment, allowing users to interact with people and objects in the real world without removing the virtual reality device or navigating a series of user interface (UI) controls or menus.
[0063] Partial perspective systems allow users to view real-world video when a triggering event occurs to initiate partial video perspective. Examples of triggering events include: when the user makes a specific gesture, when the user's hand leaves a designated area, when the user's hand is within a threshold distance of a real-world object, when the user's hand is within a threshold distance of a physical surface, and combinations thereof. Partial video perspective can provide a video feed of a portion of the user's actual environment, overlaid on the virtual environment the user is viewing. Therefore, if a user needs to interact with nearby objects or other people in the area, the user can trigger an event to access video perspective to view the user's environment in real time. The following combines... Figure 5 Block 502 in the document provides additional details about identifying the triggering event.
[0064] In some cases, users can specify a 'task area' for interaction within the virtual environment (e.g., an area on a table). Real-world objects can be cleared from the task area to prevent collisions between user actions and real-world objects. This can include placing objects outside the task area. However, users may want to interact with these objects occasionally (e.g., take a sip of coffee). In this case, triggering events could include the user placing their hand across a virtual barrier defining the task area, or moving their hand within a threshold distance of the tabletop or another real-world object. A partial perspective system can respond to this triggering event by automatically activating partial video perspective, allowing the user to see and interact with a portion of the real world while otherwise maintaining the user's experience within the virtual reality environment. The following combines... Figure 6 Additional details on how to use the task area are provided.
[0065] Cameras on virtual reality devices and / or in the user's environment can be used to capture video to be overlaid on the virtual environment to provide partial video perspective. In various implementations, the partial perspective system can also use input from one or more sensors attached to the user's (one or more) hands and / or determine the position of the user's hands by analyzing the captured video to determine the hand's position in the environment. The partial perspective system can then select a portion of the captured video based on the user's hand position and overlay that portion onto the virtual environment. The following combines... Figure 5 Blocks 504 to 508 provide additional details for capturing video feeds, identifying a portion, aligning that portion with a portion of the virtual reality environment, and adding that portion as an overlay based on that alignment.
[0066] In various implementations, the selected portion of the captured video and the corresponding coverage area in the virtual reality environment can take various forms. In some implementations, referred to herein as a "halo" implementation, the selected portion can be the area around the user's hand that triggered the event. The following is combined with... Figure 6 Additional details are provided for the halo-like implementation. In other implementations, referred to herein as the "flashlight" implementation, the selected area can be an region defined by projecting lines from the user's hand onto a real-world surface. The following combines... Figure 7 Additional details are provided for the flashlight-style implementation.
[0067] Embodiments of the disclosed technology may include artificial reality systems, or may be implemented in combination with artificial reality systems. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user. This artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with captured content (e.g., real-world photographs). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that provides a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or for use in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that deliver artificial reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, mobile devices or computing systems, "cave" environment projection systems or other projection systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.
[0068] As used herein, “virtual reality” or “VR” refers to an immersive experience where the user’s visual input is controlled by a computing system. “Augmented reality” or “AR” refers to a system in which a user views images of the real world after they have passed through a computing system. For example, a tablet computer with a camera on its back can capture images of the real world and then display those images on a screen on the side of the tablet opposite the camera. The tablet computer can process and adjust or “enhance” the images as they pass through the system, such as by adding virtual objects. “Mixed reality” or “MR” refers to a system in which light entering the user’s eyes is partly generated by a computing system and partly constitutes light reflected from objects in the real world. For example, an MR headset can be shaped as a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, thus allowing the MR headset to present virtual objects that are blended with the real objects that the user can see. As used in this article, “artificial reality,” “surreal,” or “XR” refers to any one of the following: VR, AR, MR, or any combination or mixture thereof.
[0069] Existing XR systems provide virtual spaces. However, current XR systems lack a mechanism for providing video perspective overlays on a portion of the virtual environment. Instead, users must navigate through a series of menus or options to access the full video perspective of the virtual environment (e.g., showing only the video), deactivate the virtual reality device, or remove it. This can lead to users losing progress on completed tasks or becoming distracted, resulting in decreased productivity. While working in a virtual environment, users may also risk collisions with objects in the real-world environment (“user environment”). For example, while working in a virtual environment, a user might accidentally bump into a water bottle or coffee cup, causing a spill that could damage computing devices, including virtual reality computing systems. In another example, a user might urgently need to speak with someone entering the room, and may not have time to navigate menus to access the full video perspective, deactivate the virtual reality device, or remove it.
[0070] Various aspects of this invention provide improvements to the functionality of existing virtual reality computing systems. Cameras and / or sensors are used to identify an area within the user environment indicated by the user's hand. A video stream of the indicated area is then provided to the user as an overlay of the virtual environment the user is viewing, via a virtual reality device, allowing the user to simultaneously see both the virtual environment and the indicated area in the real world. Displaying the video stream as an overlay allows the user to perform both tasks within and outside the virtual environment (e.g., interacting with objects or other people in the user environment) without needing to disable and / or remove the virtual reality device.
[0071] One embodiment of the disclosed technology utilizes a camera and / or sensors to detect the position of a user's hand within a dedicated workspace in the user environment. As the user's hand moves across the dedicated workspace, the camera and / or sensors detect the position of the user's hand within the user environment. Then, video of the area surrounding the user's hand is provided as an overlay of the virtual environment the user is viewing in the virtual reality device, allowing the user to view and interact with objects outside the dedicated workspace within the user environment.
[0072] A second embodiment of the disclosed technology utilizes a camera and / or sensors to detect a user's hand. Then, for example, image recognition technology or other techniques are used to detect the user's gesture. When a gesture is detected, the camera and / or sensors detect the area in the user's environment that the gesture is pointing to. Video of that area is provided as an overlay of the virtual environment via a virtual reality device.
[0073] Several embodiments are described in more detail below with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating an overview of devices on which certain embodiments of the disclosed technology may operate. These devices may include hardware components of a computing system 100 that provides a partially perspective video feed to a virtual reality device. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing devices 101, 102, and 103) that communicate via wired or wireless channels to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted device capable of providing a user with a computer-created or enhanced experience without the need for external processing or sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted device and a core processing unit (such as a console, mobile device, or server system), wherein some processing operations are performed on the head-mounted device while other processing operations are offloaded to the core processing unit. The following is combined with… Figure 2A and Figure 2B An example head-mounted device is described. In some embodiments, location and environmental data may be collected solely by sensors incorporated in the head-mounted device, while in other embodiments, one or more of a plurality of non-head-mounted computing devices may include sensor components capable of tracking environmental or location data.
[0074] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which may be located in one device or distributed across multiple devices (e.g., distributed across two or more computing devices 101 to 103).
[0075] The computing system 100 may include one or more input devices 120 that provide input to the processor 110, thereby notifying the processor of actions. These actions may be transmitted by a hardware controller that interprets signals received from the input devices and transmits the information to the processor 110 using a communication protocol. Each input device 120 may, for example, include a mouse, keyboard, touchscreen, touchpad, wearable input device (e.g., haptic gloves, bracelets, rings, earrings, necklaces, watches, etc.), camera (or other light-based input device, such as an infrared sensor), microphone, or other user input device.
[0076] Processor 110 may be coupled to other hardware devices, for example, via an internal bus or an external bus, such as a Peripheral Component Interconnect Standard (PCI) bus, a Small Computer System Interface (SCSI) bus, or a wireless connection. Processor 110 may communicate with a hardware controller for various devices, such as for display 130. Display 130 may be used to display text and graphics. In some embodiments, display 130 includes an input device, which is part of the display, for example, when the input device is a touchscreen or equipped with an eye orientation monitoring system. In some embodiments, the display is separate from the input device. Examples of display devices are liquid crystal displays (LCDs), light-emitting diode (LED) displays, projection displays, holographic displays, or augmented reality displays (such as heads-up displays or head-mounted displays). Other input / output (I / O) devices 140 may also be coupled to the processor. Other I / O devices include network chips or network cards, video chips or video cards, audio chips or audio cards, Universal Serial Bus (USB), FireWire or other external devices, cameras, printers, speakers, optical disc read-only memory (CD-ROM) drives, digital video disc (DVD) drives, disk drives, etc.
[0077] The computing system 100 may include a communication device capable of wirelessly or via a wired connection to other local computing devices or network nodes. This communication device may, for example, communicate with another device or server over a network using Transmission Control Protocol / Internet Protocol (TCP / IP). The computing system 100 may use this communication device to distribute operations across multiple network devices.
[0078] Processor 110 can access memory 150, which may be contained within one of a plurality of computing devices of computing system 100, or may be distributed across a plurality of computing devices of computing system 100, or a plurality of other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard disk drive, floppy disk, optical disk drive (CD), DVD, magnetic storage devices, and magnetic tape drives. Memory is not a propagation signal detached from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160, which stores programs and software such as operating system 162, partial perspective system 164, and other application programs 166. The memory 150 may also include a data memory 170, which may include virtual reality environment data, configuration data, settings, user options or preferences, etc., that can be provided to the program memory 160 or any element of the computing system 100.
[0079] Some implementations can operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be used with this technology include, but are not limited to: XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices.
[0080] Figure 2AThis is a line drawing of a virtual reality head-mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a strap 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, multiple locators 225, and one or more computing units 230. The position sensors 220, IMU 215, and computing units 230 may be located inside the HMD 200 and may be invisible to the user. In various embodiments, the IMU 215, position sensors 220, and locators 225 may track the movement and position of the HMD 200 in the real world and virtual environments with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locators 225 may emit infrared beams that produce light spots on real objects around the HMD 200. As another example, IMU 215 may include, for example, one or more accelerometers, gyroscopes, magnetometers, other non-camera-based position, force, or orientation sensors, or combinations thereof. One or more cameras (not shown) integrated with HMD 200 can detect light spots. The computing unit 230 in HMD 200 can use the detected light spots to infer the position and movement of HMD 200, and to identify the shape and position of real objects around HMD 200.
[0081] The electronic display 245 may be integrated with the front rigid body 205 and may provide image light to the user as instructed by the computing unit 230. In various embodiments, the electronic display 245 may be a single electronic display or multiple electronic displays (e.g., one display per user eye). Examples of the electronic display 245 include: liquid crystal display (LCD), organic light-emitting diode (OLED) display, active-matrix organic light-emitting diode display (AMOLED), display comprising one or more quantum dot light-emitting diode (QOLED) subpixels, projector unit (e.g., micro LED, laser, etc.), some other display, or some combination thereof.
[0082] In some implementations, HMD 200 may be coupled to a core processing unit such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor HMD 200 (e.g., via light emitted from HMD 200), and the PC may use the HMD in conjunction with outputs from IMU 215 and position sensor 220 to determine the position and movement of HMD 200.
[0083] Figure 2B The diagram shows a line drawing of a mixed reality head-mounted display (HMD) system 250, which includes a mixed reality head-mounted display (HMD) 252 and a core processing unit (CPU) 254. The HMD 252 and CPU 254 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 256. In other embodiments, the mixed reality system 250 may include only a head-mounted display without external computing devices, or may include other wired or wireless connections between the HMD 252 and the CPU 254. The HMD 252 includes a see-through display 258 and a frame 260. The frame 260 may accommodate various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye-tracking sensors, microelectromechanical systems (MEMS) components, network components, etc.
[0084] A projector can be coupled to a perspective display 258, for example, via optical elements, to display media to a user. The optical elements may include one or more waveguide components, reflectors, lenses, mirrors, collimators, gratings, etc., for directing light from the projector to the user's eyes. Image data can be transmitted from the core processing unit 254 to the HMD 252 via link 256. A controller in the HMD 252 can convert the image data into multiple light pulses from the projector, which can be transmitted as output light to the user's eyes via the optical elements. This output light can be mixed with light passing through the display 258, thereby allowing the output light to present virtual objects that appear as if they exist in the real world.
[0085] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc. The aforementioned motion and position tracking units, cameras, light sources, etc. allow HMD system 250 to track itself, for example in 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), render virtual objects as if they were stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.
[0086] Figure 2C A controller 270 is shown. In some embodiments, a user can hold the controller with one or both hands to interact with the artificial reality environment presented by HMD 200 and / or HMD 250. The controller 270 can communicate with the HMD directly or via an external device (e.g., core processing unit 254). The controller may have its own IMU unit, position sensors, and / or can emit light spots at greater distances. HMD 200 or 250, external sensors, or sensors within the controller can track these controller light spots to determine the controller's position and / or orientation (e.g., tracking the controller in 3DoF or 6DoF). The computing unit 230 in HMD 200 or core processing unit 254 can combine IMU outputs and position outputs, using this tracking to monitor the position and movement of the user's hands. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A-274B) that a user can actuate to provide input and interact with objects.
[0087] In various implementations, the HMD 200 or 250 may also include additional subsystems, such as eye-tracking units, audio systems, various network components, etc., to monitor user interactions and indications of intent. For example, in some implementations, instead of a controller or in addition to a controller, one or more cameras included in or from external cameras in the HMD 200 or 250 may monitor the position and posture of the user's hands to determine gestures and other hand movements and body movements.
[0088] Figure 3 This is a block diagram illustrating an overview of an environment 300 on which some embodiments of the disclosed technology may run. Environment 300 may include one or more client computing devices 305A to 305D, examples of which may include computing system 100. In some embodiments, some of the multiple client computing devices (e.g., client computing device 305B) may be HMD 200 or HMD system 250. The multiple client computing devices 305 may operate in a network environment using logical connections via network 330 to one or more remote computers (such as server computing devices).
[0089] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of those requests through other servers (such as servers 320A to 320C). Server computing devices 310 and 320 (also referred to as servers 310 and 320) may include computing systems such as computing system 100. Although each server computing device 310 and 320 is logically presented as a single server, multiple server computing devices can be a distributed computing environment that comprises multiple computing devices located in the same physical location or in geographically different physical locations.
[0090] Client computing device 305, and server computing devices 310 and 320 can each act as a server or client for one or more other server / client devices. Server 310 can connect to database 315. Servers 320A to 320C can each connect to their respective databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and these servers can share a database or have their own database. Although databases 315 and 325 are logically presented as a single unit, both databases 315 and 325 can be distributed computing environments containing multiple computing devices, located within their respective servers, or located in the same physical location or in geographically different physical locations.
[0091] Network 330 can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 can be the Internet or some other public or private network. Client computing device 305 can be connected to network 330 via a network interface such as via wired or wireless communication. Although the connection between server 310 and the multiple servers 320 is shown as a separate connection, these connections can be any type of LAN, WAN, wired network, or wireless network, including network 330 or a separate public or private network.
[0092] Figure 4This diagram illustrates a component 400 that may be used in a system employing the disclosed technology in some embodiments. Component 400 may be included in a single device of computing system 100 or may be distributed across multiple devices of computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, systems implementing the disclosed technology may use various hardware, including a processing unit 412, working memory 414, input and output devices 416 (e.g., a camera, display, IMU unit, network connection, etc.), and storage memory 418. In various embodiments, storage memory 418 may be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, storage memory 418 may be one or more hard disk drives or flash drives accessible via a system bus, or it may be accessed by a cloud storage provider (such as in storage 315 or 325) or other network storage, which may be accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (such as client computing device 305) or on a server computing device (such as server computing device 310 or 320).
[0093] Middleware 420 may include components that facilitate resource transfer between hardware 410 and multiple dedicated components 430. For example, middleware 420 may include an operating system, a service system, a driver, a basic input output system (BIOS), controller circuitry, or other hardware or software systems.
[0094] Multiple dedicated components 430 may include software or hardware configured to perform operations for providing partial video perspective in a virtual reality environment. The multiple dedicated components 430 may include: a sensor and camera module 434, a virtual environment module 436, an image recognition module 438, a partial perspective module 440, and component and application programming interface (API) (such as interface 432) that can be used to provide a user interface, transmit data, and control the multiple dedicated components. In some embodiments, component 400 may reside in a computing system distributed across multiple computing devices, or it may be an interface to a server-based application that executes one or more of the multiple dedicated components 430. Although the multiple dedicated components 430 are depicted as separate components, these dedicated components may be logical functional distinctions or other non-physical distinctions, and / or may be submodules or code blocks of one or more applications.
[0095] The sensor and camera module 434 can receive data from one or more sensors, such as radar sensors, lidar sensors, ultrasonic sensors, other types of range sensors, depth sensors, and other types of sensors, and / or one or more cameras. This data is then provided to one or more other modules among a plurality of dedicated components 430. In some embodiments, the sensor and camera module 434 can perform one or more functions to format any raw received data from the one or more sensors and / or cameras into a format that the other dedicated components 430 can use to perform the corresponding function of each component. The following is combined with... Figure 5 Block 502 in the document provides additional details on data acquired from sensors and / or cameras.
[0096] The virtual environment module 436 can provide a virtual environment to a user (e.g., a user of a virtual reality device such as HMD 200). In various embodiments, the virtual environment displays virtual objects. For example, the virtual environment may include product or logo designs, blueprints, social platforms, games, work applications, or any other type of content. The virtual environment module can also allow manipulation of virtual objects. For example, in various embodiments, a user can zoom in or out of the virtual environment, switch between multiple different views of the same or different virtual environments, select and manipulate (e.g., position, resize, modify, etc.) virtual objects in the virtual environment, and perform other operations that manipulate the virtual environment. In some embodiments, the virtual environment module 436 provides access to the virtual environment to the partial perspective module 440 and receives from the partial perspective module 440 video data to be overlaid on the virtual environment at a location defined by the partial perspective module 440.
[0097] The image recognition module 438 can receive video data from the sensor and camera module 434, and can analyze the video data using one or more image recognition algorithms and / or machine learning models to detect the user's hand and / or other objects in the user's real-world environment. The image recognition module 438 can provide the partial perspective module 440 with any identified objects and their positions in the user's real-world environment. (The following is a continuation of the previous paragraph.) Figure 5 Blocks 502 to 504 in the document provide additional details for identifying objects from sensor and video data.
[0098] The partial perspective module 440 can receive access to the virtual environment from the virtual environment module 436, the position of any identified object in the user's real-world environment from the image recognition module 438, and sensor data and / or camera data from the sensor and camera module 434. The partial perspective module 440 can use these inputs to determine what video data should be displayed as a partial perspective video feed. The video data to be displayed as a partial perspective video feed is based on the identified object (hand, person, or other object) and the size of the desired perspective area (as referenced below). Figure 5 The hand perspective video feed is determined as described in block 504. Once the hand perspective video feed is determined, the partial perspective module 440 determines its position within a virtual environment, which corresponds to the position of the hand perspective video feed in the user's real-world environment. The partial perspective module 440 then transmits the hand perspective video feed to a virtual environment module 436, which displays the hand perspective video feed at a location in the virtual environment that corresponds to the position of the identified object in the user's real-world environment. The following is in conjunction with... Figure 5 Blocks 504 to 508 in the middle provide additional details that offer partial video perspective.
[0099] Those skilled in the art will understand that the above description Figures 1 to 4 The components shown in each of the flowcharts discussed below can be modified in various ways. For example, the order of logic can be rearranged, multiple sub-steps can be executed in parallel, the logic shown can be omitted, and other logic can be included. In some implementations, one or more of the components described above can perform one or more processes described below.
[0100] Figure 5 A flowchart illustrating a process 500 for providing partial video perspective to a virtual reality device in some embodiments is provided. In some embodiments, process 500 may begin when the virtual reality device is powered on and run as a background process while the user operates the virtual reality device. In other embodiments, process 500 begins based on a specific context. For example, process 500 may only begin when the user is sitting at a desk or other workstation or when the user actuates an input mechanism. In some embodiments, the input mechanism may be a physical input mechanism, such as a button or switch on an HMD 200. In other embodiments, the input mechanism may be a software input mechanism, such as the selection of elements of a graphical user interface or the selection of options from a series of menus.
[0101] At block 502, process 500 detects a hand perspective trigger event. In some embodiments of this technology, the hand perspective trigger event is detected based on input from one or more cameras and / or sensors (such as depth sensors, infrared sensors, LiDAR sensors, and other types of sensors). In some embodiments of this technology, the hand perspective trigger event may be the detection of a user's hand leaving a dedicated workspace, such as a defined area on a desktop, a defined three-dimensional space in front of the user, or another workspace. For example, the workspace may be defined as a volume on the user's desktop with specific dimensions, such as three feet long, two feet wide, and one foot high. When the user's hand is within the workspace, the virtual reality system displays a virtual environment. When one of the user's hands leaves the defined workspace (as detected by a camera and / or one or more sensors), process 500 detects this event as a hand perspective trigger event. In other implementations, a hand perspective trigger event is detected if the hand leaves the workspace and is within a threshold distance (e.g., 3 inches, 6 inches, 1 foot, etc.) from the table surface, but no hand perspective event is detected if the user's hand leaves the workspace beyond the threshold distance from the workspace surface.
[0102] In other embodiments of this technology, the hand-viewing trigger event is a gesture made by the user. For example, a user may make a gesture such as: pointing a fist outward from the user's body, pointing an open palm outward from the user's body, clenching the hand like holding a flashlight and pointing it outward from the user's body, pointing the index finger in a direction and pointing the thumb upward ("pistol" gesture), making a pinching motion with two or more fingers and / or the thumb, pointing to a location with only one finger or thumb, or another gesture. When the camera detects this gesture using image recognition technology, a hand-viewing trigger event is detected.
[0103] In another embodiment of this technology, process 500 may detect a hand perspective trigger event based on a prediction that a collision is imminent between the user's hand (or other body part) and an object in the user's environment. For example, one or more cameras and / or sensors may identify that the user's hand is about to collide with a water bottle on a table. One or more cameras and / or sensors may use kinematic models to track hand speed and trajectory to determine whether the hand is about to collide with an object. If a potential collision is detected, process 500 may detect the hand perspective trigger event.
[0104] In some implementations of this technology, process 500 can detect a hand perspective trigger event based on the user's hand being within a threshold distance (e.g., 2 inches, 6 inches, 8 inches, etc.) of a real-world object. For example, one or more of the camera and / or sensors can recognize that the user's hand has moved within 6 inches of the coffee cup, thereby causing process 500 to detect the trigger event.
[0105] In some other embodiments of this technology, the hand perspective trigger event may be a user-activated control (such as a button or switch on a virtual reality device or associated controller), or it may be detected in response to a voice command from the user detected by one or more microphones.
[0106] At block 504, process 500 identifies the hand perspective video feed. In some embodiments of this technology, process 500 identifies a perspective area around the user's hand to allow partial video perspective. For example, the perspective area around the user's hand may be a "halo" or other circular or elliptical shape around the hand, or an area defined by an offset from the outer contour of the user's hand, allowing the user to view objects within the immediate vicinity of the hand. In other embodiments, the perspective area may be a "flashlight," or a projected circle or ellipse, which allows the user to view objects within the user environment based on where the user's hand is pointing. In still other embodiments, process 500 may identify only specific objects to be overlaid in the virtual environment, such as objects that the user's hand may potentially collide with or objects located near the user's hand in the user environment.
[0107] Process 500 identifies the hand perspective video feed by using image recognition technology to identify the control hand and / or (one or more) trigger objects. Once the hand or object is identified, process 500 determines the size of the perspective area. The size of the perspective area can be fixed or variable. If the size is fixed, it can be preset "out of the box" or before the user first operates the virtual reality system, or it can be set as a user option. If the size is variable, it can be set using the ambient background. For example, the size of the perspective area can be determined based on the distance between the hand and one or more objects to be shown in the perspective area, and can increase or decrease as the user's hand approaches (one or more) objects. For example, the radius of a flashlight perspective circle projected onto a real-world surface can be proportional to the distance between the user's hand and the surface. The size of the perspective area can also be controlled by one or more gestures. For example, the user can use both hands to make an extending or widening gesture to increase the size of the area or the radius of the flashlight perspective circle, or the size of the halo area around the user's hand can be proportional to the distance between the user's hands.
[0108] If the hand perspective video feed is a flashlight perspective video feed, process 500 projects light from the user's hand in one direction (such as straight out, curved, downward, or straight down). A real-world object (such as a coffee cup or a person) is used to identify the point of contact, and the hand perspective video feed is acquired from the area surrounding that point of contact. If the hand perspective video feed is triggered when the user's hand approaches and touches an object, the hand perspective video feed can be acquired from the video of that object and optionally from the video of an area within a threshold distance around that object.
[0109] At block 506, process 500 aligns the hand perspective video feed of the perspective area with the virtual environment. Process 500 accomplishes this by matching the coordinates associated with the hand perspective video feed with coordinates in the virtual environment. In some embodiments, this includes matching the hand perspective video feed to a determined position of the user's hand in the virtual environment. Process 500 can use data from a combination of one or more cameras and one or more sensors to determine the position of the user's hand in the user environment. Next, process 500 can determine the position within the virtual environment corresponding to the position of the hand or object in the user environment. In other embodiments, process 500 can match the position of the object to be perspectived and overlaid on the virtual environment in the user environment with a corresponding position in the virtual environment. Process 500 can accurately align the hand perspective video feed in the virtual environment based on the matching of the position of the user's hand or object in the user environment with its position within the virtual environment. By aligning the hand perspective video feed with the virtual environment, the hand perspective video feed allows the user to have a depth-accurate view of the user environment to aid in interaction with objects. Process 500 continues to track the position of the hand and / or object, and continuously provides perspective video feeds of the hand in different parts of the virtual environment.
[0110] At block 508, process 500, based on the alignment performed in block 506, overlays a hand perspective video feed (determined in block 504) onto the virtual environment. The hand perspective video feed is provided within the perspective area, allowing the user to interact with the object and / or other people. The portion of the virtual environment that would normally be displayed in the perspective area is replaced by the hand perspective video feed. The hand perspective video feed can be presented unmodified or partially opaque, allowing the user to "see through" the virtual environment while being able to view portions of the real world. In some implementations, the edges (one or more) of the hand perspective video feed can be "faded out" or gradually transitioned into the virtual environment using video effects.
[0111] Figure 6A conceptual diagram is provided for an example 600 of a halo-like partial video perspective overlaid on a virtual environment 610 in some embodiments employing the disclosed technology. In example 600, a user's hand 615 passes through a virtual wall 620 of the virtual environment 610 at a height 625 below a defined threshold. As described at block 502, a hand perspective trigger event is detected when the hand 615 passes through the virtual wall 620 of the virtual environment at a height 625 below the defined threshold. Then, as described at block 504, process 500 determines what will be included in the hand perspective video feed, such as the size of the perspective halo 605. In example 600, the size of the halo is defined as an area offset from both: A) the outline of the hand 615 and B) an object 630, within a threshold distance of the hand 615. (See reference...) Figure 5 As described in block 506, process 500 aligns a hand perspective video feed within a virtual environment by receiving data from one or more cameras and / or sensors. Process 500 then displays objects 630 (such as a coffee cup) from the user's real-world environment overlaid on the virtual environment 610 in a partially perspective halo 605.
[0112] Figure 7 A conceptual diagram is provided for example 700 of a flashlight-style partial video perspective 705 (also referred to as flashlight-style partial perspective 705) overlaid on a virtual environment 715 in some embodiments employing the disclosed technology. In example 700, one or more sensors and / or cameras detect gestures of a user's hand 710. The user's hand gesture 710 is detected as a hand perspective trigger event. When a gesture is detected, process 500 determines the location (shown by the flashlight-style partial perspective 705) in the user's real-world environment. For example, as referenced... Figure 5 As discussed in block 504, process 500 can identify the point of contact of a "ray" extending from the user's hand 710 and provide a video feed of the location of that point of contact and a video feed of the area surrounding that point of contact as a hand perspective video feed. In example 700, a real-world person exists in the area pointed to by the user. Process 500 aligns the hand perspective video feed with the corresponding position in the virtual environment. (See reference...) Figure 5 As described in block 506, process 500 aligns the hand perspective video feed by identifying the position of the contact point within the user environment and then determining the corresponding position within the virtual environment. For example, the position of the contact point in the user environment may correspond to a point on a virtual wall in the virtual environment 715. Then, process 500 overlays the hand perspective video feed onto the corresponding position of the contact point in the virtual environment 715, thereby displaying the portion of the user environment that the user is pointing at within the flashlight-style partial perspective 705.
[0113] In this specification, references to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of this disclosure. The appearance of these expressions in different places in the specification does not necessarily refer to the same embodiment, nor is it necessarily a separate or alternative embodiment that is mutually exclusive with other embodiments. Furthermore, various features that may be presented by some embodiments but not by others are described. Similarly, various requirements are described, which may be requirements for some embodiments but not for others.
[0114] As used herein, above the threshold means that the value of the comparison item is higher than other specified values, the comparison item is among a specified number of items with a maximum value, or the comparison item has a value within a specified highest percentage value. As used herein, below the threshold means that the value of the comparison item is lower than other specified values, the comparison item is among a specified number of items with a minimum value, or the comparison item has a value within a specified lowest percentage value. As used herein, within the threshold means that the value of the comparison item is between two other specified values, the comparison item is among a specified number of items in the middle, or the comparison item has a value within a specified percentage range in the middle. When not otherwise defined, relative terms such as high or unimportant can be understood as assigning a value and determining how that value is compared to an established threshold. For example, the phrase "select fast connections" can be understood as selecting connections with assigned values corresponding to connection speeds above the threshold.
[0115] As used herein, the word “or” refers to any possible permutation of a set of items. For example, the expression “A, B, or C” means at least one of the following: A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiples of any items (such as A and A; B, B, and C; A, A, B, C, and C, etc.).
[0116] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as exemplary forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited beyond the scope of the appended claims.
[0117] All patents, patent applications, and other references mentioned above are incorporated herein by reference. Modifications may be made to various aspects to provide further implementations using the systems, functions, and concepts of the aforementioned references. In the event of any conflict between statements or subject matter in the incorporated references and statements or subject matter of this application, this application shall prevail.
Claims
1. A method for providing a partially transparent video to a user of a virtual reality device, the method comprising: Display a computer-generated virtual environment to the user; Detect hand perspective trigger events based on user actions and / or conditions occurring in the user's real-world environment; In response to detecting the hand perspective trigger event, identify hand perspective video feeds from one or more cameras of the virtual reality device that capture the user's real-world environment; By matching the positions of real-world objects and / or the user's real-world hand positions with corresponding positions in the virtual environment, the hand perspective video feed is aligned with a portion of the virtual environment; and A perspective video feed of the hand, including the user's real-world hand and / or the real-world object, is dynamically overlaid on the aligned portion of the virtual environment, wherein the dynamic overlay provides the user with a depth-accurate view of: a) the user's real-world hand and / or b) the real-world object via the perspective video feed of the hand.
2. The method according to claim 1, wherein, The detection of the hand perspective trigger event includes: recognizing a specific gesture made by the user based on the monitored hand posture of the user, wherein the hand posture of the user is monitored using input from one or more sensors attached to the user's hand or wrist, and / or monitored by analyzing images acquired by the virtual reality device.
3. The method according to claim 1, wherein, The detection of the hand perspective trigger event includes: recognizing that the user's hand has left the designated area.
4. The method according to claim 1, wherein, The detection of the hand perspective trigger event includes: determining that the user's hand is within a threshold distance from the real-world surface.
5. The method according to claim 1, wherein, The detection of the hand perspective trigger event includes: determining that the user's hand is within a threshold distance of a real-world object, or wherein the detection of the hand perspective trigger event includes: identifying a predicted collision between a part of the user and a real-world object by analyzing the user's kinematic model using the determined current hand speed and current hand trajectory.
6. The method according to claim 1, wherein, The detection of the hand perspective trigger event includes: determining that the user speaks a specific voice command or presses a specific button associated with the virtual reality device.
7. The method according to claim 1, in, The detection of the hand perspective trigger event is based on the user's hand movements; and The identification of the hand perspective video feed includes: Identify the user's hand performing the action in one or more images from one or more cameras of the virtual reality device; and Identify the area around the identified hand in one or more images from one or more cameras of the virtual reality device.
8. The method according to claim 1, wherein, Identifying the hand perspective video feed includes: Determine the point where a specific straight line or curve extending from the user's hand intersects a real-world object or surface; and Identify the area surrounding the determined point in one or more images from one or more cameras of the virtual reality device.
9. The method according to claim 1, in, The detection of the hand perspective trigger event is based on the identification of an object in one or more images from one or more cameras of the virtual reality device; and The identification of the hand perspective video feed includes: identifying the area around the identified object in one or more images from one or more cameras of the virtual reality device.
10. The method according to claim 1, in, The detection of the hand perspective trigger event is based on the identification of an object or surface in one or more images from one or more cameras of the virtual reality device; and The size of the hand perspective video feed is set based on a defined distance between the user's hand and the identified object or surface.
11. The method according to claim 1, wherein, Identifying the hand perspective video feed includes: Identify the user's first hand and the user's second hand in one or more images from one or more cameras of the virtual reality device; and Identify an area around the identified first hand in one or more images from one or more cameras of the virtual reality device, wherein the size of the area is based on a determined distance between the first hand and the second hand.
12. The method according to claim 1, wherein, Aligning the hand perspective video feed with the virtual environment includes: matching coordinates associated with the hand perspective video feed with coordinates in the virtual environment based on the matching of one or more identified objects and / or body parts between the hand perspective video feed and the virtual environment.
13. A computing system for providing partially transparent video to a user of a virtual reality device, the computing system comprising: One or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform the method according to any one of claims 1 to 12 or to perform a process comprising the following steps: Display a computer-generated virtual environment to the user; Detect hand perspective trigger events based on user actions and / or conditions occurring in the user's real-world environment; In response to detecting the hand perspective trigger event, identify hand perspective video feeds from one or more cameras of the virtual reality device that capture the user's real-world environment; By matching the positions of real-world objects and / or the user's real-world hand positions with corresponding positions in the virtual environment, the hand perspective video feed is aligned with a portion of the virtual environment; and A perspective video feed of the hand, including the user's real-world hand and / or the real-world object, is dynamically overlaid on the aligned portion of the virtual environment, wherein the dynamic overlay provides the user with a depth-accurate view of: a) the user's real-world hand and / or b) the real-world object via the perspective video feed of the hand.
14. The computing system according to claim 13, wherein, The detection of the hand perspective trigger event includes: determining that the user's hand is within a threshold distance of a real-world object or a real-world surface.
15. The computing system according to claim 13, wherein, The detection of the hand perspective trigger event includes: recognizing a specific gesture made by the user based on the monitored hand posture of the user, wherein the hand posture of the user is monitored using input from one or more sensors attached to the user's hand, wrist, or forearm, and / or monitored by analyzing images acquired by the virtual reality device.
16. The computing system according to claim 13, in, The detection of the hand perspective trigger event is based on the user's hand movements; and The identification of the hand perspective video feed includes: Identify the user's hand performing the action in one or more images from one or more cameras of the virtual reality device; and Identify the area around the identified hand in one or more images from one or more cameras of the virtual reality device.
17. The computing system according to claim 13, wherein, Detecting the hand perspective trigger event includes: recognizing that the user's hand has left the designated area.
18. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform the method according to any one of claims 1 to 12 or to perform a process comprising the following steps: Display a computer-generated virtual environment to the user; Detect hand perspective trigger events based on user actions and / or conditions occurring in the user's real-world environment; In response to detecting the hand perspective trigger event, identify hand perspective video feeds from one or more cameras of the virtual reality device that capture the user's real-world environment; By matching the positions of real-world objects and / or the user's real-world hand positions with corresponding positions in the virtual environment, the hand perspective video feed is aligned with a portion of the virtual environment; and A perspective video feed of the hand, including the user's real-world hand and / or the real-world object, is dynamically overlaid on the aligned portion of the virtual environment, wherein the dynamic overlay provides the user with a depth-accurate view of: a) the user's real-world hand and / or b) the real-world object via the perspective video feed of the hand.
19. The computer-readable storage medium of claim 18, wherein, Identifying the hand perspective video feed includes: Determine the point where a specific straight line or curve extending from the user's hand intersects a real-world object or surface; and Identify the area surrounding the determined point in one or more images from one or more cameras of the virtual reality device.
Citation Information
Patent Citations
Spatial relationships for integration of visual images of physical environment into virtual reality
CN108780357A
