Systems and methods for virtual reality immersive calling

Abstract

1. A system for immersive virtual reality communication comprising: a first capture device configured to capture an image stream of a first user; a first network configured to transmit the captured image stream of the first user; a second network configured to receive data based at least in part on the captured image stream of the first user; and a first virtual reality device used by a second user, wherein the first virtual reality device is configured to render a virtual environment and generate a rendition of the first user based at least in part on the data based at least in part on the image stream of the first user generated by the first capture device.

Description

[Technical field]

[0001] The present invention relates to virtual reality, and more particularly to a method and system for immersive virtual reality communication. [Background technology]

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority from U.S. Provisional Patent Applications Nos. 63 / 295,501 and 63 / 295,505, both filed on December 30, 2021, the entireties of which are incorporated by reference herein.

[0003] Given the huge advances being made in virtual or mixed reality, it is becoming practical to participate in virtual meetings or social gatherings and be able to see each other in 3D faces in real time using headsets or head-mounted displays (HMDs). In some scenarios, such as pandemics and other disease outbreaks, the need for such gatherings becomes more important as people are unable to meet in person.

[0004] However, the images of different users used in a virtual environment are often taken with different devices and at different positions and angles. These inconsistent user positions / poses and lighting conditions have a significant impact on participants' ability to have a fully immersive virtual meeting experience. Summary of the Invention

[0005] According to one embodiment, a system is provided for immersive virtual reality communication, the system including a first capture device configured to capture an image stream of a first user, a first network configured to transmit the captured image stream of the first user, a second network configured to receive data based at least in part on the captured image stream of the first user, a first virtual reality device used by a second user, a second capture device configured to capture an image stream of the second user, and a second virtual reality device used by the first user, wherein the first virtual reality device is configured to render a virtual environment and generate a rendition of the first user based at least in part on the data based at least in part on the image stream of the first user generated by the first capture device, and the second virtual reality device is configured to render a virtual environment and generate a rendition of the second user based at least in part on data based at least in part on the captured image stream of the second user generated by the second capture device.

[0006] In some embodiments, the virtual environment is substantially common between the first virtual reality device and the second virtual reality device, and a viewpoint of the first virtual reality device is different from a viewpoint of the second virtual reality device. In other embodiments, the virtual environment may provide a common feel, but may be selectively configured based on viewpoints of individual users. In a further embodiment, the system further includes instructing the first user and the second user to move and rotate to optimize a position of the first user and a position of the second user relative to the first capture device and the second capture device, respectively, based on a desired rendered environment before renditions of the first user and the second user are generated via a user interface at each of the second virtual reality devices and the second virtual reality devices.

[0007] According to yet another embodiment, the first network includes at least one graphics processing unit, and the data based at least in part on the captured image stream of the first user is generated entirely in the graphics processing unit before being transmitted to the second network. [Brief description of the drawings]

[0008] These and other objects, features, and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments thereof, when considered in conjunction with the accompanying drawings and the appended claims. [Figure 1] FIG. 1 is a diagram illustrating a virtual reality capture and display system. [Diagram 2] FIG. 2 is a diagram illustrating an embodiment of a system in which two users are in two respective user environments according to the first embodiment. [Diagram 3] FIG. 3 illustrates an example of a virtual reality environment as rendered to a user. [Figure 4] FIG. 4 is a diagram illustrating an example of a second virtual perspective 400 of the second user 270 of FIG. [Figure 5A] , [Figure 5B] 5A and 5B are diagrams illustrating an example of an immersive call in a virtual environment from the perspective of a user starting position. [Figure 6] FIG. 6 is a flow chart illustrating a call initiation flow that places first and second users of the system in the proper positions to implement desired call characteristics. [Figure 7] FIG. 7 is a diagram showing an example of directing a user to a desired location. [Figure 8] FIG. 8 is a diagram showing an example in which a person is detected via a capture device and a skeleton is estimated as three-dimensional points. [Figure 9A] , [Figure 9B] , [Figure 9C] , [Figure 9D] , [Figure 9E] , [Figure 9F] 9A-9E show exemplary user interfaces for user-recommended actions of moving right, moving left, moving backward, moving forward, turning left, and turning right, respectively. [Figure 10] FIG. 10 is a diagram showing the overall flow of an immersive virtual call according to an embodiment of the present invention. [Figure 11] FIG. 11 is a diagram illustrating an example of a system for a virtual reality immersive communication system. [Figure 12A] , [Figure 12B] , [Figure 12C] , [Figure 12D] 12A-D show a user workflow for adjusting a sitting or standing position in an immersive calling system. [Figure 13A] , [Figure 13B] , [Figure 13C] , [Figure 13D] , [Figure 13E] 13A-E show various embodiments for boundary setting in an immersive communication system. [Figure 14] , [Figure 15] , [Figure 16] , [Figure 17] , [Figure 18] 14, 15, 16, 17 and 18 show various scenarios for user interaction in an immersive call system. [Figure 19] FIG. 19 is a diagram showing an example of converting an image displayed by a game engine in a GPU. [Figure 20] FIG. 20 illustrates a wireless version of FIG. [Figure 21] FIG. 21 shows a version of FIG. 19 that uses a stereo camera for capture. [Figure 22]FIG. 22 shows an example workflow for region-based object relighting using the Lab color space. [Figure 23] FIG. 23 illustrates a region-based method for object or environment relighting using the Lab color space, using the human face as an example. [Figure 24] FIG. 24 shows a user wearing a VR headset. [Figure 25A] , [Figure 25B] 25A and 25B show an example of 468 facial feature points extracted from both the input and target images. [Figure 26] FIG. 26 shows a region-based method for relighting an object or environment using the covariance matrix of the RGB channels.

[0009] Throughout the figures, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the present disclosure will now be described in detail with reference to the figures, it is done so in connection with illustrative exemplary embodiments. It is intended that changes and modifications can be made to the exemplary embodiments described without departing from the true scope and spirit of the subject disclosure as defined by the appended claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Please note that the following exemplary embodiments are merely examples for implementing the present disclosure, and can be appropriately modified or changed according to the individual configuration and various conditions of the device to which the present disclosure is applied. Therefore, the present disclosure is not limited to the following exemplary embodiments, and according to the figures and embodiments described below, the described embodiments can be applied / implemented in situations other than those described below as examples. Furthermore, when multiple embodiments are described, each embodiment can be combined with each other unless expressly specified otherwise. This includes the ability to substitute various steps and functions between embodiments as deemed appropriate by those skilled in the art. FIG. 1 shows a virtual reality capture and display system 100. The virtual reality capture system includes a capture device 110. The capture device may be, for example, a camera with a sensor and optics designed to capture 2D RGB images or videos. Some embodiments use specialized optics that capture multiple images from different viewpoints, such as a binocular camera or a light field camera. Some embodiments include one or more such cameras. In some embodiments, the capture device may include a range sensor that effectively captures RGBD (red, green, blue, depth) images directly or through software / firmware fusion of multiple sensors, such as RGB and range sensors (e.g., a lidar system or a point cloud-based depth sensor). The capture device may be connected to local or remote (e.g., cloud-based) systems 150 and 140 (hereafter referred to as server 140) via a network 160, respectively. The capture device 110 is configured to communicate with the server 140 via the network connection 160, and the capture device transmits a sequence of images (e.g., a video stream) to the server 140 for further processing. Also shown in FIG. 1 is a user of the system 120. In an exemplary embodiment, the user is wearing a virtual reality (VR) device 130 configured to transmit stereo video to the left and right eyes of the user 120.As an example, the VR device may be a headset worn by a user. Other examples may include a stereoscopic display panel or any display device that allows for the implementation of the embodiments described in this disclosure. The VR device is configured to receive incoming data from the server 140 via a second network 170. In some embodiments, the network 170 may be the same physical network as the network 160, but the data transmitted from the capture device 110 to the server 140 may be different from the data transmitted between the server 140 and the VR device 130. Some embodiments of the system do not include the VR device 130, as described below. The system may also include a microphone 180 and a speaker / headphone device 190. In some embodiments, the microphone and speaker device are part of the VR device 130.

[0011] 2 illustrates an embodiment of a system 200 with two users 220 and 270 in two separate user environments 205 and 255. In this exemplary embodiment, each of the users 220 and 270 is equipped with a separate capture device 210 and 260, a separate VR device 230 and 280, and is connected to a server 250 via a separate network 240 and 290. In some examples, only one user may have a capture device 210 or 260, while the other user may only have a VR device. In this case, one user environment may be considered a sender and the other user environment may be considered a receiver with respect to video capture. However, in embodiments where the roles of the sender and receiver are different, audio content may be transmitted and received by only or both the sender and receiver, or the roles may be reversed.

[0012] FIG. 3 illustrates a virtual reality environment 300 rendered to a user. The environment includes a computer graphic model 320 of a virtual world with a computer graphic projection of a captured user 310. For example, the user 220 of FIG. 2 can view the virtual world 320 and a rendition 310 of a second user 270 of FIG. 2 through a separate VR device 230. In this example, the capture device 260 captures images of the user 270, processes them on the server 250, and renders them in the virtual reality environment 300. In the example of FIG. 3, the user rendition 310 of the user 270 of FIG. 2 shows the user without a separate VR device 280. Some embodiments show the user with a VR device 280. In other embodiments, the user 270 does not use a wearable VR device 280. Additionally, in some embodiments, the captured image of the user 270 captures a wearable VR device, but the processing of the user image removes the wearable VR device and replaces it with a portrait of the user's face.

[0013] Additionally, the addition of the user rendition 310 to the virtual reality environment 300 along with the VR content 320 may include a lighting adjustment step to adjust the lighting of the captured and depicted user 310 to better match the VR content 320.

[0014] In this disclosure, a first user 220 of Figure 2 is shown the VR rendition 300 of Figure 3 via a separate VR device 230. Thus, the first user 220 sees the user 270 and the virtual environment content 320. Similarly, in some embodiments, a second user 270 of Figure 2 is in the same VR environment 320 but from a different perspective, such as the perspective of a virtual character rendition of 310.

[0015] Figure 4 shows a second virtual perspective 400 of the second user 270 of Figure 2. The second virtual perspective 400 is shown on the virtual device 230 of the first user 220 of Figure 2. The second virtual perspective may be based on the same virtual content 320 of Figure 3, but includes virtual content 420 from the perspective of a virtual rendition of the character 310 of Figure 3 that represents the point of view of the user 220 of Figure 2. The second virtual perspective may also include a virtual rendition of the second user 270 of Figure 2.

[0016] FIG. 6 illustrates a call initiation flow that positions a first and second user of the system in the right position to perform the desired call characteristics, as in the two examples shown in FIG. 5A and FIG. 5B. The flow begins in block B610 with a first user initiating an immersive call to a second user. The call may be initiated via an application on the user VR device or through other intermediate devices such as the user local computer, mobile phone, voice assistant (e.g., Alexa, Google Assistant, Siri, etc.). The call initiation executes instructions that notify a server, such as shown in 140 of FIG. 1 or 250 of FIG. 2, that the user intends to make an immersive call with the second user. The first user may be selected, via an app, from a list of contacts, for example, that are known to have immersive calling capabilities. The server responds to the call initiation in block B620 by notifying the second user that the first user intends to initiate an immersive call with the second user. As just a few examples, an application on the second user's local device, such as the user's VR device, cell phone, computer, or voice assistant, provides a final notification to the second user giving the second user an opportunity to accept the call. If the call is not accepted at block B630, either by the second user's selection or via a timeout period waiting for an answer, flow proceeds to block B640 and the call is rejected. In the call, the first user may be notified that the second user did not actively or passively accept the call. Other embodiments include when the second user is detected to be either in do not disturb mode or on another active call. In these cases, the call may not be accepted. If the call is accepted at block B630, flow proceeds to block B650 for the first user and to block B670 for the second user. At blocks B650 and B670, the respective users are notified that they should wear their respective VR devices. At this point, the system begins streaming video from the respective image capture devices of the first and second users.The video stream is processed via the server to detect the presence of the first and second users and determine the location of the users in the captured image. Blocks B660 and B680 then provide cues via the VR device application for the first and second users, respectively, to move to appropriate positions for an effective immersive conversation.

[0017] The collective effect of the system is therefore to present a virtual world comprising 300 of FIG. 3 and 400 of FIG. 4 that presents the illusion of a meeting of a first user and a second user in a shared virtual environment.

[0018] 5A and 5B show two examples of immersive calling in a virtual environment from the perspective of a user starting position. For example, in some cases shown in FIG. 5A, the users' renditions 510 and 520 are positioned side-by-side. Side-by-side positioning may be preferable if, for example, both users intend to watch VR content together. For example, this may be the case when watching a live event or video or other content. In other examples, as shown in FIG. 5B, the first user's rendition 560 and the second user's rendition 570 (representing users 200 and 270, respectively, of FIG. 2) are positioned in the virtual environment so as to face each other. For example, if the intention of the immersive experience is for two users to meet, they may wish to enter the environment face-to-face.

[0019] FIG. 7 illustrates an exemplary embodiment for directing a user to a desired location. The flow can be used for both a first and a second user. The flow begins at block B710 where an image capture device providing video frames to a server is analyzed to determine if a person is present in the captured image. One such embodiment performs face detection to determine if there is a face in the image. Another embodiment uses a full-person detector. Such a detector can detect the presence of a person and estimate a "human body skeleton" that can provide some estimate of the pose of the detected person.

[0020] Block B720 determines whether a person is detected. Some embodiments may include a detector capable of detecting multiple people, but only one person is of interest for purposes of an immersive call. In the case of detection of multiple people, some embodiments alert the user that there are multiple detections and ask the user to orient the other person outside the camera's field of view. In other embodiments, the most central detected person is used, and in still other embodiments, the largest detected person may be selected. It should be understood that other detection techniques may be used. If block B720 determines that a person is not detected, flow proceeds to block B725, and in some embodiments, the user is shown streaming video of the camera in the VR device headset side-by-side with the captured video captured from the VR device headset, if available. In this way, the user can see both his camera from his perspective and the scene being captured from the capture device perspective. These images may be shown, for example, as side-by-side or picture-in-picture. Flow then proceeds back to block B710, and the detection is repeated. If block B720 determines that a person is detected, flow proceeds to block B730.

[0021] In block B730, the bounds of the VR device are obtained (if available) for the current user position. Some VR devices provide a guardian boundary to prevent the user from colliding with other real-world objects while wearing the headset and immersed in the virtual world, and can detect when the user moves near or outside the virtual boundary. VR bounds are described in more detail, for example, in connection with Figures 13A-13E. Flow then proceeds to block B740.

[0022] Block B740 determines the user's pose relative to the capture device. For example, in one embodiment shown in FIG. 8, a person 830 is detected through the capture device 810 and a skeleton 840 is estimated as a 3D point. The user's pose relative to the capture device may be the pose of the user's shoulders relative to the capture device. For example, if the left and right shoulder points 860 and 870 are estimated in 3D, respectively, a unit vector n 890 may be determined to emanate from the midpoint of the two shoulder points, be perpendicular to the line 880 connecting the two shoulder points, and be parallel to the horizontal x-axis and the depth z (optical) axis 820 of the capture device. In this embodiment, the dot product of the vector n 890 with the negative capture device axis -z axis generates the cosine of the user's pose relative to the capture device. If n and z are both unit vectors, a dot product close to 1 indicates that the user is in a position such that the shoulders are facing the camera, which is ideal for capture for a face-to-face scenario as shown in FIG. 5B. However, if the dot product is close to zero, it indicates that the user is facing to the side, which is ideal for the scenario shown in FIG. 5A. Additionally, in a side-by-side scenario, one user is captured from the right side and should be positioned to the left of the other user who is to be captured from the left side. To determine if the user is oriented so that the left or right side is captured, the depths of the two shoulder points 860 and 870 can be compared to determine which shoulder is closer in depth to the capture device. In some embodiments, other joints are used as reference joints in a similar manner, such as the hips or eyeballs.

[0023] Further, the detected skeleton in FIG. 8 may be used to determine the size of the detected person based on the estimated joint lengths. Through calibration, the joint lengths may be used to estimate the standing size of the person even if they are not completely upright. This allows the detection system to determine the physical height of the bounding box of the user if the user's height is approximately known a priori. Other reference lengths may also be utilized to estimate the user's height; for example, the size of a headset is known for a given device and varies little from device to device. Thus, the user's height may be estimated based on reference lengths such as the size of the headset when they appear together in the captured frame. Some embodiments ask the user for their height when they create a contact profile so that it can be appropriately scaled when depicted in the virtual environment.

[0024] In some embodiments, the virtual environment is a real indoor / outdoor environment captured, for example, via 3D scanning and photogrammetry. Thus, through a virtual camera that corresponds to the real 3D physical world with known dimensions and sizes, through which the world is rendered to the user, the system can place renditions of people at various locations in the environment, independent of the location of the virtual camera in the environment. Thus, to provide a realistic interactive experience, a program is needed to accurately project the view of the person captured by the real camera to the desired location and pose in the environment. This can be done by creating a person-centric coordinate frame based on skeletal joints, and the system derives a reprojection matrix.

[0025] Some embodiments display a rendition of the user on a 2D projection screen (flat or curved) that is rendered in a 3D virtual environment (possibly rendered stereoscopically or via a light field display device). In these cases, it should be noted that if the viewing angle is very different from the capture angle, the projected person no longer looks realistic; in the extreme case where the projection screen is parallel to the optical axis of the virtual camera, the user simply sees a line representing the projection screen. However, due to the flexibility of the vision system, for a range of moderate differences between the capture angle in the physical world and the virtual viewing angle in the virtual world, the second user can see the projected person as a 3D person, large and beside him. This means that both communicating users can perform a limited range of movements without breaking the other's 3D perception. The range can be quantified, and this information can be used to guide the design of various embodiments for positioning the users relative to their respective capture devices.

[0026] In some embodiments, the user's rendition is presented as a 3D mesh instead of a planar projection. Such embodiments may allow for greater flexibility in the user's range of movement, which may further impact positioning purposes.

[0027] Returning to FIG. 7, once block B740 determines the user's posture relative to the capture device, flow continues to block B750.

[0028] Block B750 determines the size and position of the user within the capture device frame. Some embodiments prefer to capture the user's entire body and determine whether the entire body is visible. Additionally, in some embodiments, an estimated bounding box of the user may be determined such that the center, height, and width of that box in the capture frame are determined. Flow then proceeds to block B760.

[0029] In block B760, an optimal position is determined. In this step, first, the estimated pose of the user relative to the capture device is compared to the desired pose given the desired scenario. Second, the bounding box of the user is compared to the ideal bounding box of the user. For example, some embodiments determine that the estimated user bounding box should not extend beyond the capture frame so that the whole body can be captured, and that there is enough margin above and below the top and bottom of the box to allow the user to move and not risk moving outside the capture device area. Third, to optimize the movement margin, the user's position should be determined (e.g., the center of the bounding box) and compared to the center of the capture area. Some embodiments also check the VR boundary to ensure that the user's current placement relative to the VR boundary provides enough margin for movement.

[0030] Some embodiments include a position score S based at least in part on one or more of the orientation pose score p, the position score x, the size score s, and the bounds score b.

[0031] The pose score may be based on the dot product of vector n 890 and vector z 820 in Figure 8. Additionally, the detected z positions 860 and 870 of the left and right shoulders, respectively, are used to transform the pose angle θ. JPEG2025504337000002.jpg25146

[0032] Therefore, one pose score can be expressed as: JPEG2025504337000003.jpg1560

[0033] Here, the above norm must take into account the periodicity of θ. For example, one embodiment may desired Define || as follows: JPEG2025504337000004.jpg10137

[0034] The position score can measure the position of the detected person within the capture device frame. An example embodiment of the position score is based at least in part on the captured person bounding box center c and the capture frame width W and height H: JPEG2025504337000005.jpg2384

[0035] The boundary score b provides a score for the user's position within the VR device boundary. In this case, given a user position (u,v) on the ground plane, position (0,0) provides the position on the ground plane such that a circle of maximum radius can be constructed that circumscribes the defined boundary. In this embodiment, the boundary score may be given as: JPEG2025504337000006.jpg1548

[0036] A total score for evaluating the user pose and position can then be given as objective J: JPEG2025504337000007.jpg16150, where λ p , λ x , λ s and λ b are weighting coefficients that give the relative weight to each score, and f is a score shaping function with parameter Γ that describes the monotonic shaping function of the scores. As an example, JPEG2025504337000008.jpg1529Here, Γ b is a positive number.

[0037] Flow then moves to block B770, where it is determined whether the user's position and pose are acceptable. If not, flow continues to block B780, where visual cues are provided to the user to assist in moving to a better position. Exemplary UIs for user-recommended actions of move right, move left, move backward, move forward, turn left, and turn right, respectively, are shown in Figures 9A, 9B, 9C, 9D, 9E, and 9F. Combinations of these may also be capable of indicating the flow for the user.

[0038] 7, once the user has been provided with visual cues, flow returns to block B710 and the process repeats. If block B770 ultimately determines that the pose and position are acceptable, flow moves to block B790 and the process ends. In some embodiments, the process continues for the duration of the immersive call, and if block B770 determines that the position is acceptable, flow skips block B780, which provides positioning cues, and returns to block B710.

[0039] The overall flow of an embodiment of an immersive call is shown in FIG. 10. The flow starts in block B1010, where a call is initiated by a first user based on a selected contact as the second user and a selected VR scenario. The flow then proceeds to block B1020, where the second user accepts the call or the call is not accepted, in which case the flow ends. If the call is accepted, the flow continues to block B1030, where the first and second users are prompted to put on their VR devices (headsets). The flow then proceeds to block B1040, where the users are individually oriented to their appropriate position and posture based on the user positions relative to the respective capture devices and the selected scenario. Once the users are in an acceptable position, the flow continues to B1050, where the VR scenario begins. During the VR scenario, the user will have the option to end the call at any time. In block B1060, the call is ended and the flow ends.

[0040] 11 illustrates an exemplary embodiment of a system for a virtual reality immersive calling system. The system 11 includes two user environment systems 1100 and 1110 that are specially configured computing devices, two respective virtual reality devices 1104 and 1114, and two respective image capture devices 1105 and 1115. In this embodiment, the two user environment systems 1100 and 1110 communicate over one or more networks 1120, which may include wired networks, wireless networks, local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), and personal area networks (PANs). In some embodiments, the devices also communicate over other wired or wireless channels.

[0041] The two user environment systems 1100 and 1110 include one or more respective processors 1101 and 1111, one or more respective I / O components 1102 and 1112, and respective storage 1103 and 1113. The hardware components of the two user environment systems 1100 and 1110 also communicate via one or more buses or other electrical connections. Examples of buses include a Universal Serial Bus (USB), an IEEE 1394 bus, a PCI bus, an Accelerated Graphics Port (AGP) bus, a Serial AT Attachment (SATA) bus, and a Small Computer System Interface (SCSI) bus.

[0042] The one or more processors 1101 and 1111 include one or more central processing units (CPUs), which may include one or more microprocessors (e.g., single-core microprocessors, multi-core microprocessors), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more digital signal processors (DSPs), or other electronic circuits (e.g., other integrated circuits). The I / O components 1102 and 1112 include communication components (e.g., graphics cards, network interface controllers) that communicate with the respective virtual reality devices 1104 and 1114, the respective capture devices 1105 and 1115, the network 1120, and other input or output devices (not shown), which may include a keyboard, a mouse, a printing device, a touch screen, a light pen, an optical storage device, a scanner, a microphone, drives, and a game controller (e.g., joystick, game pad).

[0043] The storage 1103 and 1113 include one or more computer-readable storage media. As used herein, a computer-readable storage medium includes, for example, a magnetic disk (e.g., a floppy disk, a hard disk), an optical disk (e.g., a CD, a DVD, a Blu-ray), a magneto-optical disk, a magnetic tape, and a semiconductor memory (e.g., a non-volatile memory card, a flash memory, a solid-state drive, a SRAM, a DRAM, an EPROM, an EEPROM), and other manufactured products. The storage 1103 and 1113, which may include both ROM and RAM, can store computer-readable data or computer-executable instructions. The two user environment systems 1100 and 1110 also include respective communication modules 1103A and 1113A, respective capture modules 1103B and 1113B, respective drawing modules 1103C and 1113C, respective positioning modules 1103D and 1113D, and respective user performance modules 1103E and 1113E. A module includes logic, computer-readable data, or computer-executable instructions. In the embodiment shown in FIG. 11, the module is implemented in software (e.g., assembly, C, C++, C#, Java, BASIC, Perl, Visual Basic, Python, Swift). However, in some embodiments, the module is implemented in hardware (e.g., customized circuitry), or alternatively, a combination of software and hardware. When the module is implemented at least partially in software, the software may be stored in storage 1103 and 1113. Also, in some embodiments, the two user environment systems 1100 and 1110 include additional or fewer modules, the modules are combined into fewer modules, or the modules are split into more modules. One environment system may be similar to the other environment system or may differ in terms of the inclusion or organization of modules.

[0044] Each of the capture modules 1103B and 1113B includes operations programmed to perform capture as shown in 110 in FIG. 1, 210 and 260 in FIG. 2, 810 in FIG. 8, and used in block B710 in FIG. 7 and B1040 in FIG. 10. Each of the drawing modules 1103C and 1113C includes operations programmed to perform functions described in the examples of blocks B660 and B680 in FIG. 6, block B780 in FIG. 7, block B1050 in FIG. 10, and FIGS. 9A-9F. Each of the positioning modules 1103D and 1113D includes operations programmed to perform processes described in FIGS. 5A and 5B, B660 and B680 in FIG. 6, FIGS. 7, 8, and 9. Each of the user directing modules 1103E and 1113E includes operations programmed to perform user directing such as those shown in Figures 3, 4, 5A and 5B.

[0045] In another embodiment, the user environment systems 1100 and 1110 are embedded in the VR devices 1104 and 1114, respectively. In some embodiments, the modules are stored and executed on an intermediate system, such as a cloud server.

[0046] 12A-D show a user workflow for aligning a sitting or standing position between user A and user B, as described in blocks B660 and B680 of FIG. 6. In FIG. 12A, user A in a sitting position talks to user B, and the system prompts user B in a standing position to put on a headset and sit in a designated area. As a result, FIG. 12B shows a virtual meeting taking place in a sitting position. In another scenario described in FIG. 12C, user A in a standing position talks to user B. The system prompts user B in a sitting position to put on a headset and sit in a designated area. As a result, FIG. 12D shows a situation in which a virtual meeting takes place in a standing position.

[0047] Next, Figures 13, 14, 15, 16, 17 and 18 show various scenarios for user interaction in the immersive call system described herein.

[0048] Fig. 13A shows an example configuration of room-scale boundaries in a skybox setting in a VR environment. In this example, the chairs of users A and B face the same direction; Fig. 13B shows an example configuration of room-scale boundaries in a park setting in a virtual environment. In this example, the chairs of users A and B face each other and the cameras are positioned opposite each other.

[0049] 13C shows another example of setting VR boundaries, where stationary boundaries for user A and user B and corresponding room-scale boundaries are shown.

[0050] Fig. 13D shows another example configuration of the room-scale boundary in a beach setting in a VR environment. In this example, the chairs of users A and B face the same direction; Fig. 13E shows an example configuration of the room-scale boundary in a train setting in a VR environment. In this example, the chairs of users A and B face each other and the cameras are positioned opposite each other.

[0051] 14 and 15 show a VR setup of various embodiments where a side profile of the user is captured by a capture device.

[0052] Fig. 16 shows an example of performing an immersive VR activity (e.g., table tennis) where a first user faces another user and a camera is placed directly in front to capture the user's front view. This allows the user's gaze to be fixed on the other user in the VR environment. Fig. 18 shows an example of two users standing in a sports venue, where each user's profile is captured by their capture device.

[0053] In order to capture a proper image of the user, the user is prompted to move to an appropriate position and posture. Figure 17 shows an example in which the user puts on the HMD and the HMD provides instructions to the user to move their physical chair to a specified position and in a direction opposite so that a proper image of the user can be captured.

[0054] Below, we describe an embodiment for capturing an image from a camera, applying arbitrary code on the GPU to transform the image, and sending the transformed image to a game engine for display without leaving the GPU. An example is described in more detail in connection with FIG.

[0055] Advantages of this capture method include: a single copy function from CPU memory to GPU memory, all operations are performed on the GPU, the GPU's high parallel processing power allows images to be processed much faster than using the CPU, textures are shared with the game engine without leaving the GPU, making data transmission to the game engine more efficient, and reducing the time from image capture to display in the game engine application.

[0056] In the example shown in Figure 19, when a camera is connected to an application, the camera transfers frames of a video stream captured by the camera to the application to facilitate displaying the frames via a game engine. In this example, the video data is transferred to the application via an audio / video interface, such as an HDMI-USB capture card capable of capturing uncompressed video frames at high resolution with low latency. In another embodiment, as shown in Figure 20, the camera wirelessly transmits the video stream to a computer where it is decoded.

[0057] Next, the system acquires frames in the camera provided in a native format. In this step, the system acquires frames in the native format provided by the camera. In this embodiment, for the purpose of explanation only, the native format is a YUV format. The use of the YUV format is not considered limiting, and any native format that allows the implementation of this embodiment is applicable.

[0058] The data is then loaded into the GPU, and the YUV encoded frames are loaded into the GPU memory, allowing highly parallel operations to be performed on the YUV encoded frames. Once the images are loaded into the GPU, they are converted from YUV format to RGB to allow for additional downstream processing. A mapping function is then applied to remove image distortions created by the camera lens. Deep learning techniques are then employed to separate the subject from the background to remove the subject's background. To send the images to the game engine, GPU texture sharing is used to allow textures to be written to memory that the game engine reads. This process prevents data from being copied from the CPU to the GPU. The game engine is used to receive the textures from the GPU and display them to users on various devices. Any game engine that allows the implementation of the present embodiment is applicable.

[0059] In another embodiment, a stereoscopic camera is used and lens correction is performed for each half of the image, as shown in Figure 21. The stereoscopic camera displays the image captured from the left lens only to the left eye and the image captured from the right lens only to the right eye, thus providing the user with a 3D effect of the image. This can be achieved by the use of a VR headset. Any VR headset that allows the implementation of this embodiment is applicable.

[0060] Relighting of objects or environments can be very important in augmented VR. Generally, images of the virtual environment and images of different users are taken at different times and in different locations. These differences in location and time make it impossible to maintain perfectly identical lighting conditions between the users and the environment.

[0061] Different lighting conditions change the way images captured from a subject appear. Humans can use the difference in appearance to extract the lighting conditions of the environment. If different subjects are captured under different lighting conditions and then synthesized into a VR image without any processing, users will notice some inconsistencies in the lighting conditions extracted from the different subjects, which will cause an unnatural perception of the VR environment.

[0062] In addition to the lighting conditions, the cameras used for image capture for different users, as well as the virtual environments, are often different. Each camera has its own non-linear color correction function that depends on the hardware. Different cameras will have different color correction functions. This difference in color correction will also cause different lighting appearance perception for different subjects in the same lighting environment.

[0063] Given all of these lighting and camera differences and variability, in augmented VR it is important to relight the raw capture images of different subjects to make the lighting information they provide match each other.

[0064] 22 shows a workflow diagram for implementing a region-based object relighting technique using the Lab color space according to an exemplary embodiment. The Lab color space is provided as an example, and any color space that enables the implementation of the present embodiment is applicable.

[0065] Given an input image 2201 and a target image 2202, first, feature extraction algorithms 2203 and 2204 are applied to identify feature points in the target image and the input image. Then, in step 2205, a shared reference region is determined based on the feature extraction. After that, the shared region is transformed from RGB color space to Lab color space (e.g., CIE Lab color space) in steps 2206, 2207 and 2208, respectively, in both the input image and the target image. The Lab information obtained from the shared region will be used to determine a transformation matrix in steps 2209 and 2210. Then, this transformation matrix is ​​applied to the whole or part of the specific region of the input image to adjust the Lab components in step 2211, and outputs the final relighting of the input image after being converted to RGB color space in step 2212.

[0066] To explain each step in detail, an example is provided in connection with FIG.

[0067] Fig. 23 shows the workflow of the process of the present embodiment for relighting an input image based on illumination and color information from a target image. The input image is shown in A1 and the target image is shown in B1. The aim of the present invention is to make the illumination of the face in the input image closer to the illumination of the target image. First, a face is extracted using a face detection application. Any face detection application that allows the implementation of the present embodiment can be applied.

[0068] In this embodiment, the entire face from the two images was not used as a reference for relighting. The entire face was not used because in a VR environment, the user usually wears a head-mounted display (HMD), as shown in FIG. 24. As FIG. 24 shows, when the user wears the HMD, the HMD usually blocks the entire upper half of the user's face and makes only the lower half of the user's face visible to the camera. Another reason that the entire face was not used is that the content of the two faces may be different even if the user is not wearing an HMD. For example, the faces in FIG. 23(A1) and FIG. 23(A2) have an open mouth, and the faces in FIG. 23(B1) and FIG. 23(B2) have a closed mouth. The difference in the mouth region between these images may result in inaccurate adjustments to the input image if the face region should be used, or if the entire face region should be used. Also, not using the entire face provides flexibility with respect to relighting the subject. The region-based approach allows for different control to be provided for different regions of the subject.

[0069] As shown in A2 and B2 of Figure 23, a common region was selected from the lower right region of the face, shown as a rectangle in A2 (2310) and B2 (2320), and replotted as A3 and B3. The selected region served as a reference region used for relighting the input image. However, any region of the image can be selected that allows the implementation of the present invention.

[0070] Although the above description describes manual selection of specific regions for the input image and the target image, in another exemplary embodiment, the selection can be determined automatically based on feature points detected from the face. An example is shown in Fig. 25A and Fig. 25B. Application of a facial feature identifier application results in the identification of 468 facial feature points for both images. Any facial feature identifier application that allows the implementation of the present embodiment can be applied. These feature points serve as a guideline for the selection of shared regions for relighting. Any facial region can then be selected. For example, the entire lower face region can be selected as the shared region, which is indicated via the borders A and B.

[0071] After the shared region is obtained, relighting of the face in the input image is performed. The first step of this process is to convert the existing RGB color space. There are many color spaces available to represent colors, but the RGB color space is the most typical color space. However, the RGB color space is device-dependent, and different devices produce different colors. Therefore, it is not ideal to serve as a framework for color and lighting adjustment, and conversion to a device-independent color space provides better results.

[0072] As mentioned above, in this embodiment, we use the CIELAB or Lab color space. It is device independent and is calculated from the XYZ color space by normalizing to the white point. "The CIELAB color space expresses any color using three values: L*, a*, and b*. L* indicates the perceived lightness, and a* and b* can express the four colors specific to human vision" (https: / / en.wikipedia.org / wiki / CIELAB_color_space).

[0073] The Lab components from the RGB color space can be obtained, for example, by open source computer vision color transforms. After the Lab components of the shared reference regions in both the input image and the target image, their mean and standard deviation are calculated. Some embodiments use other measures of centrality and variation other than the mean and standard deviation. For example, the median and median absolute deviation can be used to robustly estimate these measures. Of course, other means are possible and this description is not limited to these only. Then, the following formula is implemented to adjust the Lab components of all or some specific selected regions of the input image: JPEG2025504337000009.jpg21161Here, x is one of the three components L^*, a^*, or b^* in CIELAB space.

[0074] In another exemplary embodiment, which is more data-driven, the covariance matrix of the RGB channels is used. By whitening the covariance matrix, it is possible to separate the RGB channels, similar to what is done using a Lab color space, such as the CIELab color space. The detailed steps are shown in Figure 26.

[0075] In FIG. 26, a feature extraction algorithm (2630 and 2640) is used to identify key feature points in both the input image and the target image. A shared reference region is then determined based on the location of the key points in both images. FIG. 26 differs from FIG. 22 in that the RGB channels are not transformed to the lab color space. Instead, the covariance matrices of the shared regions in both the input image and the target image are calculated directly from the RGB channels. Single value decomposition (SVD) is then applied to obtain a transformation matrix from these two covariance matrices. The transformation matrix is ​​applied to the entire input image to obtain the corresponding relighting matrix that is finally used and applied to correct the image displayed to the user.

[0076] At least some of the devices, systems, and methods described above may be implemented, at least in part, by providing one or more computer-readable media containing computer-executable instructions for performing the operations described above to one or more computing devices configured to read and execute the computer-executable instructions. The system or device performs the operations of the above-described embodiments when executing the computer-executable instructions. Also, an operating system on one or more of the systems or devices may implement at least some of the operations of the above-described embodiments.

[0077] Furthermore, some embodiments use one or more functional units to implement the above-described devices, systems, and methods. The functional units may be implemented in hardware only (e.g., customized circuitry) or in a combination of software and hardware (e.g., a microprocessor running software).

[0078] Furthermore, some embodiments of the devices, systems and methods combine features from two or more of the embodiments described herein. Also, as used herein, the conjunction "or" generally refers to an inclusive "or", but may refer to an exclusive "or" if expressly indicated or if the context indicates that the "or" must be an exclusive "or".

[0079] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

Claims

1. 1. A system for immersive virtual reality communication, comprising: The system comprises: a first capture device configured to capture an image stream of a first user; a first network configured to transmit the captured image stream of the first user; a second network configured to receive data based at least in part on the captured image stream of the first user; and a first virtual reality device used by a second user; Including, the first virtual reality device is configured to render a virtual environment and generate a rendition of the first user based at least in part on the data based at least in part on the image stream of the first user generated by the first capture device; A system characterized by:

2. 2. The system of claim 1, a second capture device configured to capture an image stream of the second user; a second virtual reality device used by the first user; and Further comprising: the second virtual reality device is configured to render a virtual environment and generate a rendition of the second user based at least in part on data generated by the second capture device that is based at least in part on the captured image stream of the second user. A system characterized by:

3. 3. The system of claim 2, the virtual environment is substantially common between the first virtual reality device and the second virtual reality device; a perspective of the first virtual reality device is different from a perspective of the second virtual reality device; A system characterized by:

4. 3. The system of claim 2, instructing the first user and the second user to move and rotate to optimize a position of the first user and a position of the second user relative to the first capture device and the second capture device, respectively, based on a desired rendering environment before a rendition of the first user and the second user is generated via a user interface at the second virtual reality device and the second virtual reality device, respectively; Further comprising: A system characterized by:

5. 5. The system of claim 4, instructing the first user and the second user based on a position optimization based on at least one of a user pose, a user position, a user scale, or a virtual reality device bounds; Further comprising: A system characterized by:

6. 2. The system of claim 1, the first network includes at least one graphics processing unit; the data based at least in part on the captured image stream of the first user is generated entirely in the graphics processing unit before being transmitted to the second network. A system characterized by:

7. 2. The system of claim 1, the first capture device is a stereo camera; A system characterized by:

8. 1. A method for immersive virtual reality communication, comprising: The method comprises: Capturing a first image stream of a first user; transmitting the captured image stream of the first user over a first network; receiving data over a second network based at least in part on the first captured image stream of the first user; rendering a virtual environment to generate a rendition of the first user based at least in part on the data based at least in part on the first captured image stream of the first user; Including, A method comprising:

9. 9. The method of claim 8, Capturing a second image stream of a second user; and rendering a virtual environment to generate a rendition of the second user based at least in part on data based at least in part on the second captured image stream of the second user; Further comprising: A method comprising:

10. 10. The method of claim 9, The virtual environment is common to the rendering of the first user and the rendering of the second user, The viewpoint of the first user's presentation is different from the viewpoint of the second user's presentation. A method comprising:

11. 10. The method of claim 9, instructing the first user and the second user to move and rotate to optimize a position of the first user and a position of the second user, respectively, based on a desired rendering environment before renditions of the first user and the second user are generated via a user interface; Further comprising: A method comprising:

12. 12. The method of claim 11, instructing the first user and the second user based on a position optimization based on at least one of a user pose, a user position, a user scale, or a virtual reality device bounds; Further comprising: A method comprising:

13. 9. The method of claim 8, the data based at least in part on the first captured image stream of the first user is generated entirely in a graphics processing unit before being transmitted to the second network. A method comprising:

14. 9. The method of claim 8, the first captured image stream of the first user is a three-dimensional image; A method comprising:

15. 1. A virtual reality device for immersive communication, the device comprising: Storage units that further include: a capture module configured to capture an image stream of a user; a communications module configured to transmit the captured image stream of the user over a network; a rendering module configured to render a virtual environment; a rendition module configured to generate a rendition of the first user based at least in part on the data based at least in part on the image stream of the first user generated by the capture module; and a position module configured to instruct the user based on a position optimization based on at least one of a user pose, a user position, a user scale, or a virtual reality device bounds; having An apparatus comprising:

16. 1. A computer-readable storage device having stored thereon computer-executable instructions for causing a computing device to perform a method for immersive virtual reality communication, the instructions comprising: Capturing a first image stream of a first user; transmitting the captured image stream of the first user over a first network; receiving data over a second network based at least in part on the first captured image stream of the first user; Rendering a virtual environment and generating a rendition of the first user based at least in part on the data based at least in part on the first captured image stream of the first user; Including, A computer readable storage device comprising:

17. 17. The computer readable storage device of claim 16, Capturing a second image stream of a second user; and rendering a virtual environment to generate a rendition of the second user based at least in part on data based at least in part on the second captured image stream of the second user; Further comprising: A computer readable storage device comprising:

18. 20. The computer readable storage device of claim 17, The virtual environment is common to the rendering of the first user and the rendering of the second user, The viewpoint of the first user's presentation is different from the viewpoint of the second user's presentation. A computer readable storage device comprising:

19. 20. The computer readable storage device of claim 17, instructing the first user and the second user to move and rotate to optimize a position of the first user and a position of the second user, respectively, based on a desired rendering environment before renditions of the first user and the second user are generated via a user interface; Further comprising: A computer readable storage device comprising:

20. 20. The computer readable storage device of claim 19, instructing the first user and the second user based on a position optimization based on at least one of a user pose, a user position, a user scale, or a virtual reality device bounds; Further comprising: A computer readable storage device comprising:

21. 17. The computer readable storage device of claim 16, the data based at least in part on the first captured image stream of the first user is generated entirely in a graphics processing unit before being transmitted to the second network. A computer readable storage device comprising:

22. 17. The computer readable storage device of claim 16, the first captured image stream of the first user is a three-dimensional image; A computer readable storage device comprising:

Images