Virtual reality system

Abstract

To generate an avatar of a user with a VR headset being removed.SOLUTION: An imaging computing device generates a two-dimensional (2D) image of at least one user wearing a VR-headset on which one or more sensors are mounted by using a camera device. Movement of pupils of the user is tracked by using the one or more sensors and a headset image is removed from the generated 2D image of the user wearing the headset. In order that other user can view a VR image of the user as if the user were wearing no headset, a detected and recorded pupillary image of the user rendered by the one or more sensors of the VR-headset is stitched and overlaid on a region where the headset image has been removed from the generated 2D image of the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to methods and apparatus for capturing and processing two-dimensional (2D) and three-dimensional (3D) images.

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 62 / 642,488, filed March 13, 2019, which is incorporated by reference in its entirety. [Background technology]

[0003] Virtual reality (VR) enables highly immersive experiences, offering new ways of seeing the world and the ability to explore new environments, both real and virtual. However, sharing these experiences with others is challenging compared to physical reality, as VR headsets have difficulty creating a holistic view of the people participating in the experience. Summary of the Invention [Problem to be solved by the invention]

[0004] Part of this disconnect is alleviated by mixed reality (MR), a related medium that shares a VR user's virtual context in a two-dimensional video format, allowing other viewers to feel the user's virtual experience. While MR makes sharing easier, it poses a major obstacle to a fully engaging experience and full visibility of humans in VR, as the headset continues to block facial expressions and gaze. [Means for solving the problem]

[0005] The objects and advantages of the illustrative embodiments described hereinafter will be set forth and will be apparent from the following detailed description. Additional advantages of the illustrated embodiments will be realized and attained by the apparatus, systems, and methods particularly pointed out in the written description and claims hereof, as well as the appended drawings.

[0006] The general objective of this work is to provide a live VR chat experience. Currently, VR chat applications utilize animated avatars. The objective of the invention is to replace the avatar with a live video without a headset image of the user that is visually replaced by a 3D model of the user's head (e.g., previously recorded without the user wearing the headset). Another objective of the invention is for each user to adjust their own level of realism (e.g., from an avatar to a live 3D video image). [Brief description of the drawings]

[0007] The accompanying appendix and / or drawings depict various non-limiting, exemplary inventive aspects according to the present disclosure.

[0008] [Figure 1] 1 illustrates example computing and mobile devices that can be used to implement the techniques described herein. [Diagram 2] 1 shows a user wearing a headset and the view the user sees of other users wearing headsets. [Diagram 3] This shows the process of removing the headset image from the user to create a VR for the user, as if the headset had been removed. [Figure 4] The following illustrates some of the steps for creating a VR chat between users in the following embodiment. [Diagram 5] The following illustrates some of the steps for creating a VR chat between users in the following embodiment. [Figure 6] The following illustrates some of the steps for creating a VR chat between users in the following embodiment. [Figure 7] The following illustrates some of the steps for creating a VR chat between users in the following embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The described embodiments will be more fully described with reference to the attached drawings, in which like reference numbers identify like structural / functional features. The exemplary embodiments described below are merely exemplary and can be implemented in various forms as understood by those skilled in the art, so the exemplary embodiments are in no way limited to those illustrated. Therefore, the structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for claims and as expressions to teach those skilled in the art to use the discussed embodiments in various ways. Furthermore, the terms and phrases used herein are not intended to be limiting, but are intended to provide an understandable description of the illustrated embodiments.

[0010] Where a range of values ​​is listed, unless the context clearly dictates otherwise, the upper and lower limits of that range, as well as the stated range, are included in the exemplified embodiments. The upper and lower limits of these smaller ranges may be independently included in the smaller ranges and are included in the exemplified embodiments, subject to any specifically excluded limit in the stated range. When the stated range includes one or both of the limits, ranges excluding either of those included limits are also included in the exemplified embodiments.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this invention belongs.Although any method and material similar or equivalent to those described herein can also be used in the implementation or testing of the illustrated embodiment, exemplary methods and materials are described here.All publications mentioned herein are incorporated by reference to disclose and describe the method and / or material related to which the publication is cited.

[0012] It should be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, a reference to "a stimulus" includes a plurality of such stimuli, a reference to a "signal" includes a reference to one or more signals and equivalents thereof known to those skilled in the art, and so forth.

[0013] The publications discussed herein are provided solely for their disclosure prior to the filing date of the present application. Nothing herein should be construed as an admission that the illustrated embodiments are not entitled to antedate such publication by virtue of prior invention. Further, the publication dates provided may be different from the actual publication dates which may need to be independently confirmed.

[0014] It should be understood that the exemplary embodiments described below are preferably software algorithms, programs or code stored on a computer readable medium having control logic enabling execution on a machine having a computer processor. The apparatus typically includes a storage device configured to provide output from the execution of the computer algorithm or program.

[0015] The term "software" as used herein is meant to be synonymous with code or programs residing in a host computer's processor, regardless of whether the implementation is hardware, firmware, a software computer product available on a disk, memory storage device, or for download from a remote device. The embodiments described herein include such software for implementing the above-mentioned equations, relationships, and algorithms. Those skilled in the art will appreciate further features and advantages of the illustrated embodiments based on the above embodiments. Thus, the illustrated embodiments should not be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated herein in their entirety by reference.

[0016] Reference is now made in explanatory fashion to the drawings, in which like reference characters denote like elements throughout the several views. Figure 1 illustrates an exemplary general purpose computing system 600 and a general purpose mobile computing device 650 that may be used with the techniques described herein. The computing system 600 and the mobile device 650 should be understood as exemplary embodiments in which one or more components thereof may be used with the illustrated embodiments of the invention described herein.

[0017] Computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 connecting to memory 604 and a high-speed expansion port 610, and a low-speed bus 614 and a low-speed interface 612 connecting to storage device 606. Each of components 602, 604, 606, 608, 610, and 612 may be interconnected using various buses and may be attached to a common motherboard or otherwise, as appropriate. Processor 602 may process instructions for execution within computing device 600, including instructions stored in memory 604 or storage device 606, to display graphical information for a GUI on an external input / output device, such as a display 616 connected to high-speed interface 608. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Additionally, multiple computing devices 600 may be connected, with each device providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0018] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is a volatile memory unit or units. In other implementations, the memory 604 is one or more non-volatile memory units. The memory 604 may also be other forms of computer readable media, such as magnetic or optical disks.

[0019] The storage device 606 allows the computing device 600 to provide mass storage. In one implementation, the storage device 606 is a computer-readable medium such as a floppy disk drive, a hard disk drive, an optical disk drive, or may include a tape drive, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. The computer program product may be tangibly embodied on an information carrier. Also, the computer program product may include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium such as the memory 604, the storage device 606, or a memory on the processor 602.

[0020] The high-speed controller 608 manages the bandwidth-intensive operations of the computing device 600, and the low-speed controller 612 manages the less bandwidth-intensive operations. Such an allocation of functions is merely exemplary. In one implementation, the high-speed controller 608 is coupled to the memory 604, the display 616 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 610 that can accept various expansion cards (not shown). In such an implementation, the low-speed controller 612 is coupled to the storage device 606 and the low-speed expansion port 614. The low-speed expansion port can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and is connected to one or more input / output devices such as a keyboard, pointing device, scanner, or network device such as a switch or router, for example, via a network adapter.

[0021] The computing device 600 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 620, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 624. It may also be implemented in a personal computer, such as a laptop computer 622. Alternatively, components from the computing device 600 may be combined with other components in a mobile device (each of such devices may include one or more of the computing devices 600, 650, and the entire system may be composed of multiple computing devices 600, 650 communicating with each other).

[0022] Computing device 650 includes, among other things, a processor 652, a memory 664, input / output devices such as a display 654, a communication interface 666, and a transceiver 668. Device 650 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, and some of these components may be mounted on a common motherboard or otherwise as desired.

[0023] The processor 652 can execute instructions within the computing device 650, including instructions stored in the memory 664. The processor can be implemented as a chipset of chips including separate analog and digital processors. The processor can provide coordination of other components of the device 650, such as, for example, control of a user interface, applications run by the device 650, and wireless communications by the device 650.

[0024] The processor 652 may communicate with a user via a control interface 658 and a display interface 656 coupled to a display 654. The display 654 may be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) or other suitable display device. The display interface 656 may comprise appropriate circuitry for driving the display 654 to present graphical and other information to the user. The control interface 658 may receive commands from the user and convert them for providing to the processor 652. Additionally, an external interface 662 may be provided to communicate with the processor 652 to enable short-range communication of the device 650 with other devices. The external interface 662 may provide, for example, wired communication in some implementations, or wireless communication in other implementations, and multiple interfaces may also be used.

[0025] The memory 664 stores information within the computing device 650. The memory 664 may be implemented as one or more of a computer-readable medium or a plurality of volatile or non-volatile memory units. An expansion memory 674 may also be provided and connected to the device 650 via an expansion interface 672, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 674 may provide additional storage space for the device 650 or may also store applications or other information for the device 650. In particular, the expansion memory 674 may include instructions that perform or supplement the above steps and may also include secure information. Thus, for example, the expansion memory 674 may be provided as a security module for the device 650 and may be programmed with instructions that enable secure use of the device 650. Additionally, secure applications may be provided via the SIMM card along with additional information, such as placing identifying information on the SIMM card in a manner that cannot be hacked.

[0026] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 664, expansion memory 674, or memory on processor 652, and may be received, for example, via transceiver 668 or external interface 662.

[0027] The device 650 can communicate wirelessly via a communication interface 666, which may include digital signal processing circuitry as required. The communication interface 666 may provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via a radio frequency transceiver 668. Additionally, short-range communication is possible, such as by using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 670 may provide the device 650 with additional navigation and location-related wireless data, which may be used appropriately by applications executing on the device 650.

[0028] Device 650 can also communicate audibly using audio codec 660, which can receive speech information from a user and convert the speech information into usable digital information. Similarly, audio codec 660 can generate audible sounds for the user, such as through a speaker in a handset of device 650. Such sounds can include sounds from a voice call, but can also include recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 650.

[0029] Computing device 650 may be implemented in a number of different forms, as shown in the figure. For example, computing device 650 may be implemented as a mobile phone 680. Computing device 650 may also be implemented as part of a smartphone 682, personal digital assistant, or other similar mobile device.

[0030] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations include implementation in one or more computer programs executable and / or interpretable by a programmable system that includes at least one programmable processor that is adapted for special or general purpose input and output of data and instructions to a storage system, at least one input device, and at least one output device.

[0031] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor (including machine-readable media that receive machine instructions as machine-readable signals). The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0032] To provide for user interaction, the systems and techniques described herein can be implemented on a computer that has a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) (and other suitable display technologies) for displaying information to the user, a keyboard, and a pointing device (such as a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). Also, input from the user can be received in any form, including acoustic, speech, or tactile input.

[0033] The systems and techniques described herein can be implemented in a computing system. The computing system includes a back-end component (e.g., as a data server), or a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0034] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0035] The computing device shown in Figure 1 is interfaced with a virtual reality (VR headset 690). In the illustrated embodiment, one or more sensors are included in the VR headset to virtually remove the appearance of the headset 690 when one user interacts with another user. For example, the headset sensors 690 may include one or more LEDs 691 that illuminate the user's eyes while tracking the user's pupils, and one or more micro-cameras 693 to record the user's eyes / pupils.

[0036] For example, an embodiment of the present invention uses the camera device 400 to create a model of the user's face 704 without the headset 690 as a proxy for the user's hidden face when the headset 690 is worn. This proxy is used to synthesize the face in a mixed reality (MR) video. This gives the impression that the headset 690 is removed when interacting with other users in the VR realm. For example, this initial calibration step may require the user 704 to sit in front of the color + depth camera 400 and a monitor, and then track markers on the monitor with their eyes. This calibration procedure obtains a 3D face model of the user 704 and creates a database (via sensors 691 and 693) that maps appearance images (or textures) to different gaze directions and blinks. The aforementioned gaze database (i.e., a face model with textures, preferably indexed by gaze) allows the user's facial appearance to be dynamically changed to generate the desired (and real-time) gaze. This allows the creation of natural and lifelike synthesized facial appearances. It should be understood that the facial configuration can be modified by using appropriate wireframe software techniques.

[0037] In some implementations, one or more input devices in addition to the computing devices (e.g., mouse, keyboard) can be rendered in the computer-generated 3D environment. The rendered input devices (rendered mouse, rendered keyboard, etc.) can be used as rendered in the VR space to control objects in the VR space.

[0038] Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are for illustrative purposes only and are not intended to limit the implementation of the invention(s) described and / or claimed in this document.

[0039] Thus, an exemplary computing system environment 100 has been generally illustrated and described above with reference to Figure 1. Implementations of various exemplary embodiments will now be described with reference to Figures 2-7.

[0040] It should be understood that the environment of use of the exemplary embodiment of the present invention relates to a system 600 that allows online chat, preferably in VR using live video. It should be understood that the example discussed below used to explain the exemplary embodiment assumes a live chat between two users, a local user and a remote user (although the present invention should not be understood as being limited thereto). The exemplary embodiment includes one or more components of the system 600 described above, including a VR system such as an Oculus Rift™, one or more webcams 400 per user, and preferably a green screen. With reference to FIG. 2, during a VR chat session according to the illustrated embodiment, for example, a remote user 700 wearing a headset 690 sees live video in VR of another user 702 (wearing a headset 690) minus the headset 690 worn by a local user 704. Similarly, during the same VR chat session, a remote user 702 wearing a headset 690 sees live video in VR of the user 700 (wearing a headset 690) minus the headset 690 worn by the local user in VR.

[0041] According to one aspect of the invention (and with reference to FIG. 3), the remote user's vision of the headset is replaced by overlaying a pre-recorded still image / video of the local user's head. It is preferable to overlay only a portion of the user's head, since the mouth and lower part of the face are generally still visible when wearing a VR headset. Preferably, the live video effect is created by chromakey ("green screen") techniques, preferably using a standard 2D webcam 400 (FIG. 4). Thus, according to the illustrated embodiment, each user 704, 706 is preferably positioned in front of a portable green screen (402). It should be understood that this may be countered by certain software techniques, or that a 3D camera may be used to determine the user's contours without a green screen.

[0042] As described below, according to an exemplary embodiment, two or more users (704 and 706) can participate in a live online chat using a special webcam 400 at each user's location. Each user (704, 706) preferably wears a VR headset 690 (700, 702), whereby each user (704, 706) sees a superimposed pre-recorded image of the other user's head (706, 704). The broadcast image is preferably a composite of the live video and the pre-recorded image. The live video preferably includes the user's body up to just above the mouth area (FIG. 3) (e.g., where the VR headset 690 would typically be). Thus, the recorded image is preferably the user's nose, eyes, and top of the head (704, 706). It should be understood that live chat with more than two people can be realized, such as a corporate video conference.

[0043] This recorded imagery may also be created in advance using a specialized form of photogrammetry, a technique in which multiple 2D images are stitched together and processed to create a 3D model and texture. Currently, this process is expensive and time-consuming, requiring specialized hardware and multiple cameras. In contrast, embodiments of the present invention may utilize an inexpensive stereoscopic camera consisting of two cameras facing the user, each with either stereoscopic or monoscopic vision. For example, the user rotates in their chair while the cameras record and capture the scene. Post-processing of the stereoscopic or monoscopic camera output is used to generate the 3D model.

[0044] An exemplary process for enabling a VR session according to the illustrated embodiment will be described with reference to Figures 4-7. First, the face of each user 704 (Figure 4), 706 (Figure 5) without a headset is photographed by the webcam 400. Next, as shown in Figures 6 and 7, each user (704, 706) wears a headset 690. The headset 690 includes the aforementioned LED sensors 691 and 693 that detect gaze, movement, and blinking. The sensor 691 preferably illuminates one of the user's eyes in the headset 690 to track pupil movement while the micro-video camera sensor 693 tracks the user's pupil. The processing chip 602 is preferably configured and operable to restore the moving pupil to the face, preferably by "stitching" the pupil via software to complete the image of the face obscured by the headset 690. Thus, both users 704 and 706 can see each other's 3D images in real time (through their respective headsets 690) as if the face of the other user 704, 706 was in front of them and not obscured by the headset 690.

[0045] More preferably, by further using electro-sensitive devices placed on the forehead and eyebrows of each user, the frown expressions of each user 704, 706 may also be tracked via software processing on the processing chip 602.

[0046] In summary, according to one or more illustrated embodiments, the present invention enables a virtual projection of a user's webcam video stream in a video chat, minus the user's headset. This is accomplished by modeling the user's head in 3D, using modeling techniques such as photogrammetry. For example, the top of the user's head is replaced with this model in real time, so that other participating users only see the participating user's body, head, face, and eyes minus the headset 690.

[0047] It should be appreciated that one or more stationary stereoscopic 3D cameras (e.g., StereoLabs ZED camera, Microsoft Kinect, Intel RealSense line of 3D cameras, etc.) may be used to surround and model multiple users (e.g., three or more users). The 3D camera technology may additionally calculate the positions of the users and provide collaborative interaction in VR. For example, two users can play a game of catch on Mars or two people can take a virtual walk through the Grand Canyon together.

[0048] It should be appreciated that the computing system 600 and headset 690 of the present invention are configured and operable for use in an augmented reality (AR) setting. AR is understood to be a technology for overlaying computer-generated augmentations on top of existing reality to make it more meaningful through the ability to manipulate it. The illustrated embodiment includes AR being developed for use as an app, and preferably a headset 690 that blends digital elements into the real world in such a way that the digital elements and the real world enhance each other. For example, two or more users wearing the headset 690 of the present invention, each in a different location, can use / operate the headset 690 to appear next to each other in a selected setting (e.g., skydiving) and / or location (e.g., Paris). This allows them to not only imagine the same setting (e.g., the streets of Paris, France) in real time, but also to see each other as if the headset 690 was removed, in accordance with the teachings herein.

[0049] It should further be understood that additional embodiments of the computing system 600 and / or headset 690 may be configured to provide real-time image stitching, preferably using at least two separate video feeds and / or a single video feed, preferably in combination with a predefined still frame image cache. Since the "eye area" of the user's face is typically obscured by the VR display device, eye tracking is preferably performed via a camera mounted within the VR display device that allows real-time selection of cached and / or "mixed" images taken from an initial "setup" phase (image cache) and used for the image stitching step. This provides real-time 3D face tracking and reconstruction using the final (composite / stitched) video. It should be understood that by using a real-time digitally created face (representing the user), the face / result is preferably composited into either the real-time video and / or depth extraction of the user. It should be understood that the user's body is preferably extracted from its background using either green screen and / or depth extraction techniques. It should further be appreciated that the fully reconstructed user may be composited into numerous applications, such as, but not limited to, 360-degree video, VR chat rooms, and other applicable applications. Additionally, various technologies (e.g., SDKs, APIs, etc.) currently being used and / or researched include, but are not limited to: EmguCV, OpenCV, and MATLAB®; ZED Camera with SDK; ARKit; Primary programming languages: C# and C++; Unity (game engine); Oculus and / or Vive (VR display devices).

[0050] In the above specific exemplary embodiment, it should be understood that the various non-limiting embodiments described herein can be implemented in separate combinations or selective combinations for specific applications. Moreover, some of the various features of the above non-limiting embodiments can be used without other described corresponding features. Thus, the description provided herein should be considered as merely illustrative of the principles, teachings, and exemplary embodiments of the present invention, and not as limiting thereof. For example, the present invention can also include the following aspects: Section 1. 1. A computational method for generating a virtual reality (VR) interaction between two or more users, at least one of the users wearing a VR headset, comprising: generating a two-dimensional (2D) image of at least one user wearing a VR headset equipped with one or more sensors using a camera device; tracking pupil movement of the user using the one or more sensors; removing a headset image from the generated 2D image of the user wearing the headset; stitching the detected and recorded pupil images of the user rendered by the one or more sensors overlaid on an area of ​​the generated 2D image of the user from which the headset image has been removed such that other users can view the VR image of the user as if the user were not wearing a headset; A calculation method having the following formula: Section 2. Item 1. The computational method of item 1, wherein the 2D image is generated using a video camera. Section 3. Item 1. The method of claim 1, wherein the camera device has a depth sensor. Section 4. Item 1. The calculation method of item 1, wherein the one or more sensors include an infrared LED sensor. Section 5. Item 5. The calculation method of item 4, wherein the one or more sensors further include a micro-video camera for recording the direction of the user's pupils. Section 6. The method of claim 1, wherein users can see each other in real time. Section 7. 1. An apparatus for generating virtual reality (VR) interactions between two or more users, at least one of the users wearing a VR headset, comprising: At least one VR headset device having one or more sensor components; a camera device for capturing at least an image of a user wearing the VR headset device; a VR imaging computing device communicatively connected to each of the at least one VR headset device and a camera device; Equipped with The VR imaging calculation device includes: Memory, a processor in communication with the memory and configured to execute instructions stored in the memory; Equipped with The instruction: generating a two-dimensional (2D) image of at least one user wearing a VR headset equipped with one or more sensors using a camera device; tracking pupil movement of the user using the one or more sensors mounted in the VR headset; removing a headset image from the generated 2D image of the user wearing the headset; stitching the detected and recorded pupil images of the user rendered by the one or more sensors of the VR headset to overlay them in an area from the generated 2D image of the user where the headset image has been removed, such that other users can view the VR image of the user as if the user were not wearing a headset; An apparatus for causing the processor to execute the steps of: Section 8. Item 10. The apparatus for generating virtual reality (VR) interaction between two or more users according to item 9, wherein the camera device is a video camera. Section 9. Item 10. The apparatus for generating virtual reality (VR) interaction between two or more users according to item 9, wherein the camera device is integrated with a smartphone device. Section 10. Item 1. An apparatus for generating virtual reality (VR) interaction between two or more users as described in Item 1, wherein the camera device has a depth sensor. Section 11. Item 1, wherein the one or more sensor components mounted on the VR headset include an infrared LED sensor. Section 12. Item 12. An apparatus for generating virtual reality (VR) interaction between two or more users as described in item 11, wherein the one or more sensor components mounted on the VR headset include at least one micro video camera for recording the direction of the user's pupils. Section 13. Item 8. An apparatus for generating virtual reality (VR) interaction between two or more users as described in item 7, wherein the users can see each other in real time. Section 14. Item 8. An apparatus for generating virtual reality (VR) interaction between two or more users as described in item 7, wherein the users can view a predetermined image as a background.

Claims

1. 1. A virtual reality (VR) system, comprising: a VR headset worn by at least one user; one or more LED sensors disposed within the VR headset for tracking pupil movement of at least one user wearing the VR headset; at least one camera device disposed external to the VR headset capable of capturing 1) a video image of at least one user wearing the VR headset, and 2) one or more still images of the user not wearing the VR headset; one or more camera devices disposed within the VR headset for capturing real-time images of the user's pupils simultaneously with tracking of the user's pupil movements by the one or more LED sensors; A computer processor; Equipped with The computer processor includes: receiving from the one or more camera devices captured real-time images of the user's pupils while wearing the VR headset; receiving from the one or more LED sensors tracked pupil movement of the user's pupil while wearing the VR headset concurrently with captured real-time images of the user's pupil; generating a composite of the one or more captured still images of the user not wearing the VR headset and the captured real-time pupil image of the user captured by the one or more camera devices disposed within the VR headset; superimposing the composite on the 2D image of the user such that the composite is superimposed at a location where the image of the headset is virtually removed from the user wearing the VR headset, thereby allowing other users to view the VR image of the at least one user with the superimposed captured real-time pupil image and movement as if the at least one user were not wearing the VR headset; A VR system including instructions for:

2. The VR system of claim 1 , wherein the externally located camera device is a webcam.

3. The VR system of claim 1 , wherein the one or more LED sensors disposed within the VR headset include an infrared LED sensor.

4. The VR system of claim 1 , wherein the 2D image of the user with the superimposed pupil image and movement is rendered in real time.

Images