System and method for transmitting information for holographic communication
Through server management of sessions between user equipment and multi-dimensional communication technology, the problem of synchronous transmission of user field of view and scale data in holographic communication in augmented reality, virtual reality and mixed reality is solved, and visual coordination and Data rendering consistency improves user experience and communication efficiency.
Patent Information
- Application Number
- CN202380027082.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-12-15
- Filing Date
- 2023-03-23
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to effectively handle audio/video data between devices in augmented reality, virtual reality and mixed reality, especially in holographic communication scenarios, where there are challenges in synchronous transmission and rendering of user field of view and scale data.
Through the server, the user scale data in the audio/video data is maintained, and the user scale data is received and modified, ensuring the synchronous rendering of the user's field of view and scale in holographic communication, using multidimensional communication session management and bit rate adjustment technology. Achieve visual coordination in multi-user scenarios.
It realizes the synchronous processing of user field of view and scale data in multi-user holographic communication in augmented reality, virtual reality and mixed reality scenarios, improving user experience and communication efficiency, and ensuring visual coordination and data in multi-user scenarios Render consistency.
Smart Images

Figure CN118923119A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to wireless communications between multiple devices, including but not limited to systems and methods for signaling information for holographic communications. Background Art
[0002] Augmented reality (AR), virtual reality (VR), and mixed reality (MR) are becoming more common, with these technologies supported on a wider range of platforms and devices. Certain devices may be configured for video or audio calls and / or video or audio conferencing. Summary of the invention
[0003] Various aspects of the present disclosure are directed to systems, methods, and computer-readable media for transmitting information for holographic communications.
[0004] According to a first aspect of the present disclosure, a method is provided, comprising: maintaining, by one or more servers, a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users; receiving, by the one or more servers, audio / video (A / V) data of the first user and scale data of the first user from the first device via the first session; modifying, by the one or more servers, the scale of the first user represented in video data of the A / V data according to the scale data; and sending, by the one or more servers, the modified A / V data of the first user to one or more second devices via the one or more second sessions for rendering to the one or more second users.
[0005] In some embodiments, the method may further include: establishing, by the one or more servers, the first session and the one or more second sessions in response to a request for a three-dimensional (3D) communication session between the first device and the one or more second devices.
[0006] In some embodiments, the method may also include: receiving, by one or more servers via one or more second sessions, second A / V data of one or more second users and second scale data of one or more second users from one or more second devices; modifying, by one or more servers, the scale of one or more second users represented in the second video data of the second A / V data based on the second scale data; and sending, by one or more servers, the modified second A / V data of one or more second users to the first device via the first session for rendering to the first user of the first device.
[0007] In some embodiments, modifying the scale of the first video data may include: one or more servers modifying the scale of the user represented in the first video data based on the first scale data and the second scale data, and wherein, modifying the scale of one or more second users represented in the second video data includes: one or more servers modifying the scale of one or more second users represented in the second video data based on the first scale data and the second scale data.
[0008] In some embodiments, modifying the scale may include modifying the scale of a first user represented in the first video data to match the scale of one or more second users represented in the second video data based on the first scale data and the second scale data.
[0009] In some embodiments, the A / V data may include three-dimensional (3D) video data and spatial audio data.
[0010] In some embodiments, the method may further include receiving, by one or more servers, data indicating a field-of-view (FOV) of the first user.
[0011] In some embodiments, the data indicative of the (FOV) may be determined by the first device based on data received from a third device communicatively coupled to the first device.
[0012] In some embodiments, the method may also include: maintaining, by one or more servers, a relative position of each of the first user, the second user, and the third user in a positioning map for a first user of the first device and at least a second user of the second device and a third user of the third device; and determining, by one or more servers, the FOV of the first user based on the direction data and the relative position of the first user in the positioning map.
[0013] In some embodiments, the method may also include: one or more servers sending second A / V data of a second user to the first device at a first bit rate via the first session based on data indicating the FOV of the first user; and one or more servers sending third A / V data of a third user to the first device at a second bit rate via the first session based on data indicating the FOV of the first user.
[0014] According to another aspect of the present disclosure, one or more servers are provided. The one or more servers may maintain a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. The one or more servers may receive audio / video (A / V) data of a first user of a first device and scale data of the first user via the first session. The one or more servers may modify the scale of the first user represented in the video data of the A / V data according to the scale data. The one or more servers may send the modified A / V data of the first user to one or more second devices via one or more second sessions for rendering to one or more second users.
[0015] In some embodiments, one or more servers may establish a first session and one or more second sessions based on a request for a three-dimensional (3D) communication session between a first device and one or more second devices. In some embodiments, one or more servers may receive second A / V data of one or more second users and second scale data of one or more second users from one or more second devices via one or more second sessions. One or more servers may modify the scale of one or more second users represented in the second video data of the second A / V data based on the second scale data. One or more servers may send the modified second A / V data of one or more second users to the first device via the first session for rendering to the first user of the first device. In some embodiments, one or more servers may modify the scale of the first video data based on the first scale data and the second scale data (e.g., by modifying the scale of the user represented in the first video data), and may modify the scale of the second video data based on the first scale data and the second scale data (e.g., by modifying the scale of one or more second users represented in the second video data). In some embodiments, one or more servers may modify the scale of the first video data (e.g., the scale of the first user represented in the first video data) to match the scale of the second video data (e.g., the scale of one or more second users represented in the second video data) based on the first scale data and the second scale data.
[0016] In some embodiments, the A / V data includes three-dimensional (3D) video data and spatial audio data. In some embodiments, one or more servers may receive data indicating a field of view (FOV) of a first user. In some embodiments, the data indicating the (FOV) is determined by a first device based on data received from a third device communicatively coupled to the first device. In some embodiments, one or more servers may maintain the relative position of each of the first user, the second user, and the third user in a positioning map for a first user of the first device and at least a second user of the second device and a third user of the third device. One or more servers may determine the FOV of the first user based on the direction data and the relative position of the first user in the positioning map. In some embodiments, one or more servers may send the second A / V data of the second user to the first device at a first bit rate via a first session based on the data indicating the FOV of the first user. One or more servers may send the third A / V data of the third user to the first device at a second bit rate via a first session based on the data indicating the FOV of the first user.
[0017] According to another aspect of the present disclosure, a device is provided, which includes: one or more cameras; and one or more processors, wherein the one or more processors are configured to: determine scale data of a user of a first device based on image data collected via the one or more cameras of the first device; receive audio / video (A / V) data of the user; and send the A / V data and scale data of the user to one or more servers, so that the one or more servers modify the A / V data of the user according to the scale data, and send the modified A / V data of the user to one or more second devices.
[0018] It will be understood that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be generalizable to any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and illustrative only, and are not limitations of the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and names in the various drawings refer to like elements. For clarity, not every component is labeled in every drawing.
[0020] Figure 1 is a schematic diagram of an example wireless communication system according to an example implementation of the present disclosure.
[0021] Figure 2is a schematic diagram of a head wearable display and a console for presenting augmented reality or virtual reality according to an example embodiment of the present disclosure.
[0022] Figure 3 is a schematic diagram of a head wearable display according to an exemplary embodiment of the present disclosure.
[0023] Figure 4 is a block diagram of a computing environment according to an example implementation of the present disclosure.
[0024] Figure 5 is an example diagram of a holographic call or communication session conducted through a head-wearable device (HWD) according to an example embodiment of the present disclosure.
[0025] Figure 6 is a block diagram of a system for holographic communication according to an example embodiment of the present disclosure.
[0026] Figure 7 is a graphical representation of a virtualized map according to an example implementation of the present disclosure.
[0027] Figure 8 Included are illustrations of various views of an imaging system according to example embodiments of the present disclosure.
[0028] Fig. 9 Example images corresponding to video data that may be acquired by an imaging system according to example embodiments of the present disclosure are included.
[0029] Fig.10 is a schematic diagram of an end-user system in communication with a server according to an example embodiment of the present disclosure.
[0030] Fig.11A and Fig. 11B 2 is an example of video data frames from three different client systems before and after scaling modification according to an example embodiment of the present disclosure.
[0031] Fig.12 is a flowchart illustrating an example method for updating session conditions of a client system according to an example implementation of the present disclosure.
[0032] Fig.13 is a flow chart illustrating an example method of updating the field of view (FOV) of a device according to an example implementation of the present disclosure.
[0033] Fig.14 is a flow chart illustrating an example method of managing bit rates for objects in a communication session according to an example implementation of the present disclosure.
[0034] Fig.15is a flowchart illustrating an example method of transmitting information for holographic communication according to an example embodiment of the present disclosure.
[0035] Fig.16 is a flow chart illustrating an example method of improving field of view according to an example implementation of the present disclosure. DETAILED DESCRIPTION
[0036] Before turning to the drawings that show certain embodiments in detail, it should be understood that the present disclosure is not limited to the details or methods set forth in the specification or shown in the drawings. It should also be understood that the terminology used herein is for descriptive purposes only and should not be considered as limiting.
[0037] Figure 1 An example wireless communication system 100 is shown. The wireless communication system 100 may include a base station 110 (also referred to as a "wireless communication node 110" or "station 110") and one or more user equipment (UE) 120 (also referred to as a "wireless communication device 120" or "terminal device 120"). The base station 110 and the UE 120 may communicate via wireless communication links 130A, 130B, and 130C. The wireless communication link 130 may be a cellular communication link that complies with third generation mobile communication technology (3G), fourth generation mobile communication technology (4G), fifth generation mobile communication technology (5G), or other cellular communication protocols or Wi-Fi communication protocols. In one example, the wireless communication link 130 supports, adopts, or is based on orthogonal frequency division multiple access (OFDMA). In one aspect, the UE 120 is located within a geographical boundary relative to the base station 110 and may communicate with or through the base station 110. In some embodiments, the wireless communication system 100 includes more than one wireless communication link. Figure 1 More, fewer, or more of the components shown may be included. Figure 1 For example, the wireless communication system 100 may include components other than Figure 1 One or more additional base stations 110 other than the base station 110 shown.
[0038] In some embodiments, UE 120 may be a user device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, a wearable computing device, etc. Each UE 120 may communicate with a base station 110 via a corresponding communication link 130. For example, UE 120 may send data to base station 110 via wireless communication link 130, and receive data from base station 110 via wireless communication link 130. Example data may include audio data, image data, text, etc. Communication or data transmission by UE 120 to base station 110 may be referred to as uplink communication. Communication or data reception by UE 120 from base station 110 may be referred to as downlink communication. In some embodiments, UE 120A includes a wireless interface 122, a processor 124, a memory device 126, and one or more antennas 128. These components may be implemented as hardware, software, firmware, or a combination thereof. In some embodiments, UE 120A includes more than one embodiment. Figure 1 More, fewer, or more of the components shown may be included. Figure 1 For example, UE 120 may include an electronic display and / or an input device. For example, UE 120 may include components other than those shown. Figure 1 Additional antennas 128 and additional wireless interfaces 122 beyond the antenna 128 and wireless interface 122 shown.
[0039] Antenna 128 can be a component that receives radio frequency (RF) signals and / or sends RF signals through a wireless medium. The RF signal can be at a frequency between 200 megahertz (MHz) and 100 gigahertz (GHz). The RF signal can have a packet, symbol or frame corresponding to the data used for communication. Antenna 128 can be a dipole antenna, a patch antenna, a loop antenna or any suitable antenna for wireless communication. In one aspect, a single antenna 128 is used to both send and receive RF signals. In one aspect, different antennas 128 are used to send and receive RF signals. In one aspect, multiple antennas 128 are used to support multiple-input, multiple-out (MIMO) communications.
[0040] The wireless interface 122 includes or is implemented as a transceiver for sending and receiving RF signals through a wireless medium. The wireless interface 122 can communicate with the wireless interface 112 of the base station 110 through a wireless communication link 130A. In one configuration, the wireless interface 122 is coupled to one or more antennas 128. In one aspect, the wireless interface 122 can receive an RF signal at an RF frequency received by the antenna 128, and down-convert the RF signal to a baseband frequency (e.g., 0 GHz to 1 GHz). The wireless interface 122 can provide the down-converted signal to the processor 124. In one aspect, the wireless interface 122 can receive a baseband signal for transmission at a baseband frequency from the processor 124, and up-convert the baseband signal to generate an RF signal. The wireless interface 122 can send the RF signal through the antenna 128.
[0041] The processor 124 is a component for processing data. The processor 124 can be implemented as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a logic circuit, etc. The processor 124 can obtain instructions from the memory device 126 and execute these instructions. In one aspect, the processor 124 can receive down-converted data at a baseband frequency from the wireless interface 122, and decode or process the down-converted data. For example, the processor 124 can generate audio data or image data based on the down-converted data, and present the audio indicated by the audio data and / or the image indicated by the image data to the user of UE120A. In one aspect, the processor 124 can generate or obtain data for transmission at a baseband frequency, and encode or process the data. For example, the processor 124 can encode or process image data or audio data at a baseband frequency, and provide the encoded or processed data to the wireless interface 122 for transmission.
[0042] The memory device 126 is a component that stores data. The memory device 126 can be implemented as a random access memory (RAM), a flash memory, a read only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any device capable of storing data. The memory device 126 can be implemented as a non-transitory computer-readable medium that stores instructions that can be executed by the processor 124 to perform various functions of the UE 120A disclosed herein. In some embodiments, the memory device 126 and the processor 124 are integrated into a single component.
[0043] In some embodiments, each of UEs 120B ... 120N includes similar components of UE 120A for communicating with base station 110. Therefore, for the sake of brevity, detailed descriptions of their repeated parts are omitted herein.
[0044] In some embodiments, the base station 110 may be an evolved node B (eNB), a serving eNB, a target eNB, a femto station, or a pico station. The base station 110 may be communicatively coupled to another base station 110 or other communication device via a wireless communication link and / or a wired communication link. The base station 110 may receive data (or RF signals) from a UE 120 in an uplink communication. Additionally or alternatively, the base station 110 may provide data to another UE 120, another base station, or another communication device. Thus, the base station 110 allows communication between multiple UEs 120 associated with the base station 110, or allows communication between other UEs associated with different base stations. In some embodiments, the base station 110 includes a wireless interface 112, a processor 114, a memory device 116, and one or more antennas 118. These components may be implemented as hardware, software, firmware, or a combination thereof. In some embodiments, the base station 110 includes more than one UE 120. Figure 1 More, fewer, or more of the components shown may be included. Figure 1 For example, base station 110 may include an electronic display and / or an input device. For example, base station 110 may include Figure 1Additional antennas 118 and additional wireless interfaces 112 beyond the antenna 118 and wireless interface 112 shown.
[0045] Antenna 118 may be a component that receives radio frequency (RF) signals and / or transmits RF signals over a wireless medium. Antenna 118 may be a dipole antenna, a patch antenna, a loop antenna, or any suitable antenna for wireless communication. In one aspect, a single antenna 118 is used to both transmit and receive RF signals. In one aspect, different antennas 118 are used to transmit and receive RF signals. In one aspect, multiple antennas 118 are used to support multiple-input multiple-output (MIMO) communications.
[0046] The wireless interface 112 includes or is implemented as a transceiver for sending and receiving RF signals through a wireless medium. The wireless interface 112 can communicate with the wireless interface 122 of the UE 120 through a wireless communication link 130. In one configuration, the wireless interface 112 is coupled to one or more antennas 118. In one aspect, the wireless interface 112 can receive an RF signal at an RF frequency received by the antenna 118, and down-convert the RF signal to a baseband frequency (e.g., 0 GHz to 1 GHz). The wireless interface 112 can provide the down-converted signal to the processor 124. In one aspect, the wireless interface 122 can receive a baseband signal for transmission at a baseband frequency from the processor 114, and up-convert the baseband signal to generate an RF signal. The wireless interface 112 can send the RF signal through the antenna 118.
[0047] Processor 114 is a component for processing data. Processor 114 can be implemented as FPGA, ASIC, logic circuit, etc. Processor 114 can obtain instructions from memory device 116 and execute these instructions. In one aspect, processor 114 can receive down-converted data from baseband frequency of wireless interface 112, and decode or process the down-converted data. For example, processor 114 can generate audio data or image data according to the down-converted data. In one aspect, processor 114 can generate or obtain data for transmission at baseband frequency, and encode or process the data. For example, processor 114 can encode or process image data or audio data at baseband frequency, and provide the encoded or processed data to wireless interface 112 for transmission. In one aspect, processor 114 can set, allocate, schedule or allocate communication resources for different UE120. For example, processor 114 can set different modulation schemes, time slots, channels, frequency bands, etc. for UE 120 to avoid interference. The processor 114 may generate data (or an uplink (UL) configuration grant (CG)) indicating the configuration of communication resources and provide the data (or UL CG) to the wireless interface 112 for transmission to the UE 120 .
[0048] The memory device 116 is a component that stores data. The memory device 116 may be implemented as a RAM, a flash memory, a ROM, an EPROM, an EEPROM, a register, a hard disk, a removable disk, a CD-ROM, or any device capable of storing data. The memory device 116 may be implemented as a non-transitory computer-readable medium that stores instructions that can be executed by the processor 114 to perform various functions of the base station 110 disclosed herein. In some embodiments, the memory device 116 and the processor 114 are integrated into a single component.
[0049] In some embodiments, the communication between the base station 110 and the UE 120 is based on one or more layers of the Open Systems Interconnection (OSI) model. The OSI model may include multiple layers, including: a physical layer, a medium access control (MAC) layer, a radio link control (RLC) layer, a packet data convergence protocol (PDCP) layer, a radio resource control (RRC) layer, a non-access stratum (NAS) layer or an Internet Protocol (IP) layer, and other layers.
[0050] Figure 2is a block diagram of an example artificial reality system environment 200. In some embodiments, the artificial reality system environment 200 includes a HWD 250 worn by a user, and a console 210 that provides artificial reality (e.g., augmented reality, virtual reality, mixed reality) content to the HWD 250. Each of the HWD 250 and the console 210 may be a separate UE 120. The HWD 250 may be referred to as, include, or be part of a head mounted display (HMD), a head mounted device (HMD), a head wearable device (HWD), a head worn display (HWD), or a head worn device (HWD). The HWD 250 may detect the position and / or orientation of the HWD 250 and the shape, position and / or orientation of the user's body / hand / face, and provide the detected position and / or orientation of the HWD 250 and / or tracking information indicating the shape, position and / or orientation of the body / hand / face to the console 210. The console 210 may generate image data indicating an artificial reality image based on the detected position and / or orientation of the HWD 250, the detected shape, position and / or orientation of the user's body / hand / face, and / or user input for artificial reality, and send the image data to the HWD 250 for presentation. In some embodiments, the artificial reality system environment 200 includes the following: Figure 2 More, fewer, or more of the components shown may be included. Figure 2 In some embodiments, the functionality of one or more components in the artificial reality system environment 200 may be distributed among the components in a manner different from that described herein. For example, some of the functions of the console 210 may be performed by the HWD 250. For example, some of the functions of the HWD 250 may be performed by the console 210. In some embodiments, the console 210 is integrated as part of the HWD 250.
[0051] In some embodiments, HWD 250 is an electronic component that can be worn by a user and can present or provide an artificial reality experience to the user. HWD 250 can present one or more images, video, audio, or some combination thereof to provide an artificial reality experience to the user. In some embodiments, audio is presented via an external device (e.g., a speaker and / or headphones) that receives audio information from HWD 250, console 210, or both, and presents audio based on the audio information. In some embodiments, HWD 250 includes a sensor 255, a wireless interface 265, a processor 270, an electronic display 275, a lens 280, and a compensator 285. These components can operate together to detect the position of HWD 250 and the gaze direction of the user wearing HWD 250, and render an image of a field of view corresponding to the detected position and / or orientation of HWD 250 within an artificial reality. In other embodiments, HWD 250 includes a sensor 255, a wireless interface 265, a processor 270, an electronic display 275, a lens 280, and a compensator 285. These components can operate together to detect the position of HWD 250 and the gaze direction of the user wearing HWD 250, and render an image of a field of view corresponding to the detected position and / or orientation of HWD 250 within an artificial reality. In other embodiments, HWD 250 includes a sensor 255, a wireless interface 265, a processor 270, an electronic display 275, a lens 280, and a compensator 285. Figure 2 More, fewer, or more of the components shown may be included. Figure 2 The parts shown are different parts.
[0052] In some embodiments, sensor 255 includes electronic components, or a combination of electronic components and software components, that detect the position and orientation of HWD 250. Examples of sensor 255 may include: one or more imaging sensors, one or more accelerometers, one or more gyroscopes, one or more magnetometers, or another suitable type of sensor that detects motion and / or position. For example, one or more accelerometers may measure translational motion (e.g., forward / backward, up / down, left / right), and one or more gyroscopes may measure rotational motion (e.g., pitch, left-right sway, roll). In some embodiments, sensor 255 detects translational motion and rotational motion, and determines the orientation and position of HWD 250. In one aspect, sensor 255 may detect translational motion and rotational motion relative to a previous orientation and previous position of HWD 250, and determine a new orientation and / or new position of HWD 250 by accumulating or integrating the detected translational motion and / or rotational motion. For example, assuming that HWD 250 is facing a direction 25 degrees from a reference direction, sensor 255 may determine that HWD 250 is now facing or facing a direction 45 degrees from the reference direction in response to detecting that HWD 250 has turned 20 degrees. In another example, assuming that HWD 250 is located two feet from a reference point in a first direction, sensor 255 may determine that HWD 250 is now located at the vector product of two feet in the first direction and three feet in the second direction in response to detecting that HWD 250 has moved three feet in a second direction.
[0053] In some embodiments, the sensor 255 includes an eye tracker. The eye tracker may include an electronic component that determines the gaze direction of the user of the HWD 250, or a combination of an electronic component and a software component. In some embodiments, the HWD 250, the console 210, or a combination thereof may generate image data of artificial reality in conjunction with the gaze direction of the user of the HWD 250. In some embodiments, the eye tracker includes two eye trackers, each of which captures an image of a corresponding eye and determines the gaze direction of the eye. In one example, the eye tracker determines the angular rotation of the eye, the translation of the eye, the torsion change of the eye, and / or the shape change of the eye based on the captured image of the eye, and determines the relative gaze direction relative to the HWD 250 based on the determined angular rotation of the eye, the translation of the eye, and the torsion change of the eye. In one approach, the eye tracker may illuminate or project a predetermined reference pattern or structured pattern onto a portion of the eye, and capture an image of the eye to analyze the pattern projected onto the portion of the eye, thereby determining the relative gaze direction of the eye relative to the HWD 250. In some embodiments, the eye tracker determines the user's gaze direction in conjunction with the orientation of HWD 250 and the relative gaze direction relative to HWD 250. For example, assuming that HWD 250 is oriented in a direction 30 degrees from a reference direction and the relative gaze direction of HWD 250 is -10 degrees (or 350 degrees) relative to HWD 250, the eye tracker may determine that the user's gaze direction is 20 degrees from the reference direction. In some embodiments, the user of HWD 250 may configure HWD 250 (e.g., via user settings) to enable or disable the eye tracker. In some embodiments, the user of HWD 250 is prompted to enable or disable the eye tracker.
[0054] In some embodiments, the wireless interface 265 includes electronic components, or a combination of electronic components and software components, that communicate with the console 210. The wireless interface 265 may be or may correspond to the wireless interface 122. The wireless interface 265 may communicate with the wireless interface 215 of the console 210 via the base station 110 through a wireless communication link. Through the communication link, the wireless interface 265 may send data indicating the determined position and / or orientation of the HWD 250, and / or the determined gaze direction of the user to the console 210. In addition, through the communication link, the wireless interface 265 may receive an indication from the console 210 or image data corresponding to an image to be rendered, as well as additional data associated with the image.
[0055] In some embodiments, the processor 270 includes an electronic component, or a combination of electronic components and software components, that generates one or more images for display (e.g., according to a change in the field of view of the artificial reality space). In some embodiments, the processor 270 is implemented as a part of the processor 124 or is communicatively coupled to the processor 124. In some embodiments, the processor 270 is implemented as a processor (or a graphics processing unit (GPU)) that executes instructions to perform various functions described herein. The processor 270 can receive image data describing an artificial reality image to be rendered, and additional data associated with the image through the wireless interface 265, and can render the image to be displayed through the electronic display 275. In some embodiments, the image data from the console 210 can be encoded, and the processor 270 can decode the image data to render the image. In some embodiments, the processor 270 receives object information and depth information in the additional data from the console 210, the object information indicating virtual objects in the artificial reality space, and the depth information indicating the depth of these virtual objects (or the distance from the HWD 250). In one aspect, the processor 270 may perform shading, reprojection, and / or blending based on the artificial reality image from the console 210, object information, depth information, and / or updated sensor measurements from the sensor 255 to update the artificial reality image to correspond to the updated position and / or orientation of the HWD 250. Assuming that the user turns his head after the initial sensor measurements, the processor 270 may generate a small portion (e.g., 10%) of the image corresponding to the updated field of view within the artificial reality based on the updated sensor measurements and append the portion to the image in the image data from the console 210 by reprojection, rather than reconstructing the entire image in response to the updated sensor measurements. The processor 270 may perform shading and / or blending on the additional edges. Thus, the processor 270 may generate an artificial reality image without reconstructing the artificial reality image based on the updated sensor measurements.
[0056] In some embodiments, the electronic display 275 is an electronic component that displays an image. The electronic display 275 may be, for example, a liquid crystal display or an organic light emitting diode display. The electronic display 275 may be a transparent display that allows the user to see through. In some embodiments, when the user wears the HWD 250, the electronic display 275 is located near the user's eyes (e.g., less than 3 inches). In one aspect, the electronic display 275 emits light or projects light to the user's eyes based on the image generated by the processor 270.
[0057] In some embodiments, lens 280 is a mechanical component that changes the light received from electronic display 275. Lens 280 can magnify the light from electronic display 275 and correct optical errors associated with the light. Lens 280 can be a Fresnel lens, a convex lens, a concave lens, a filter, or any suitable optical component that changes the light from electronic display 275. Although electronic display 275 is very close to the eye, light from electronic display 275 can still pass through lens 280 to the pupil so that the user can see the image displayed by electronic display 275.
[0058] In some embodiments, compensator 285 includes electronic components, or a combination of electronic components and software components, that perform compensation to compensate for any distortion or aberration. In one aspect, lens 280 introduces optical aberrations (e.g., chromatic aberration), pincushion distortion, barrel distortion, etc. Compensator 285 can determine compensation (e.g., pre-distortion) to be applied to the image to be rendered from processor 270 to compensate for the distortion caused by lens 280, and apply the determined compensation to the image from processor 270. Compensator 285 can provide the pre-distorted image to electronic display 275.
[0059] In some embodiments, the console 210 is an electronic component or a combination of an electronic component and a software component that provides content to be rendered to the HWD 250. In one aspect, the console 210 includes a wireless interface 215 and a processor 230. These components can work together to determine the visual field of the artificial reality corresponding to the position of the HWD 250 and the gaze direction of the user of the HWD 250 (e.g., the user's field of view (FOV)), and can generate image data indicating the artificial reality image corresponding to the determined visual field. In addition, these components can work together to generate additional data associated with the image. Additional data can be information associated with presenting or rendering artificial reality in addition to the artificial reality image. Examples of additional data include hand model data, mapping information (or simultaneous localization and mapping (SLAM) data) for converting the position and orientation of the HWD 250 in the physical space to the virtual space, eye tracking data, motion vector information, depth information, edge information, object information, etc. The console 210 may provide image data and additional data to the HWD 250 for use in presenting artificial reality. Figure 2 More, fewer, or more of the components shown may be included. Figure 2 In some embodiments, console 210 is integrated as part of HWD 250.
[0060] In some embodiments, the wireless interface 215 is an electronic component, or a combination of an electronic component and a software component, that communicates with the HWD 250. The wireless interface 215 may be or may correspond to the wireless interface 122. The wireless interface 215 may be a corresponding component of the wireless interface 265 to communicate via a communication link (e.g., a wireless communication link). Through the communication link, the wireless interface 215 may receive data from the HWD 250 indicating the determined position and orientation of the HWD 250 and / or the determined gaze direction of the user. In addition, through the communication link, the wireless interface 215 may send image data describing the image to be rendered, as well as additional data associated with the artificial reality image, to the HWD 250.
[0061] The processor 230 may include or may correspond to a component that generates content to be rendered based on the position and / or orientation of the HWD 250. In some embodiments, the processor 230 is implemented as a part of the processor 124 or is communicatively coupled to the processor 124. In some embodiments, the processor 230 may be combined with the gaze direction of the user of the HWD 250. In one aspect, the processor 230 determines the field of view of the artificial reality based on the position and / or orientation of the HWD 250. For example, the processor 230 maps the position of the HWD 250 in the physical space to a position within the artificial reality space, and determines the field of view of the artificial reality space from the mapped position in the artificial reality space along the direction corresponding to the mapped orientation. The processor 230 may generate image data (the image data describes an image of the field of view of the determined artificial reality space) and send the image data to the HWD 250 via the wireless interface 215. In some embodiments, the processor 230 may generate additional data associated with the image (the additional data includes motion vector information, depth information, edge information, object information, hand model data, etc.), and send the additional data and the image data to the HWD 250 through the wireless interface 215. The processor 230 may encode the image data describing the image, and may send the encoded data to the HWD 250. In some embodiments, the processor 230 generates image data periodically (e.g., every 11 milliseconds (ms)) and provides the image data to the HWD 250.
[0062] In one aspect, the process of detecting the position of the HWD 250 and the gaze direction of the user wearing the HWD 250 and presenting the image to the user should be performed within one frame time (e.g., 11ms or 16ms). The delay between the movement of the user wearing the HWD 250 and the image displayed corresponding to the movement of the user may cause jitter, which may cause motion sickness and may reduce the user experience. In one aspect, the HWD 250 and the console 210 can prioritize the communication of AR / VR so that the delay between the movement of the user wearing the HWD 250 and the image displayed corresponding to the movement of the user can be presented within one frame time (e.g., 11ms or 16ms) to provide a seamless experience.
[0063] Figure 3 2 is a schematic diagram of HWD 250 according to an exemplary embodiment. In some embodiments, HWD 250 includes a front rigid body 305 and a belt 310. Front rigid body 305 includes electronic display 275 ( Figure 3 Not shown), lens 280 ( Figure 3 ), sensor 255, wireless interface 265 and processor 270. Figure 3 In the illustrated embodiment, the wireless interface 265, processor 270, and sensor 255 are located within the front rigid body 305 and may not be visible to the outside. In other embodiments, the HWD 250 has a Figure 3 For example, the wireless interface 265, the processor 270 and / or the sensor 255 may be located in a different configuration than that shown. Figure 3 The positions shown are at different locations.
[0064] Various operations described herein can be implemented on a computer system. Figure 4 A block diagram of a representative computing system 414 that can be used to implement the present disclosure is shown. In some embodiments, the source device 110, the sink device 120, the console 210, and the HWD 250 are implemented by the computing system 414. The computing system 414 can be implemented as, for example, a consumer device, such as a smart phone, other mobile phones, tablet computers, wearable computing devices (e.g., smart watches, glasses, head wearable displays), desktop computers, laptop computers; or the computing system can be implemented using distributed computing devices. The computing system 414 can be implemented to provide VR experience, AR experience, MR experience. In some embodiments, the computing system 414 may include conventional computer components, such as a processor 416, a storage device 418, a network interface 420, a user input device 422, and a user output device 424.
[0065] The network interface 420 can provide a connection to a wide area network (e.g., the Internet) to which a wide area network (WAN) interface of the remote server system is also connected. The network interface 420 can include a wired interface (e.g., Ethernet) and / or a wireless interface that implements various RF data communication standards, such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, 5G, 60GHz, Long Term Evolution (LTE), etc.).
[0066] The network interface 420 may include a transceiver to allow the computing system 414 to send data using a transmitter and receive data from a remote device using a receiver. The transceiver may be configured to support transmit / receive that supports industry standards for implementing bidirectional communications. An antenna may be attached to the transceiver housing and electrically coupled to the transceiver. Additionally or alternatively, a multi-antenna array may be electrically coupled to the transceiver so that multiple beams pointing in different directions may facilitate sending and / or receiving data.
[0067] The transmitter may be configured to wirelessly transmit a frame, time slot, or symbol generated by the processor unit 416. Similarly, the receiver may be configured to receive a frame, time slot, or symbol, and the processor unit 416 may be configured to process the frame. For example, the processor unit 416 may be configured to determine the type of frame and process the frame and / or fields of the frame accordingly.
[0068] User input device 422 may include any device (or devices) via which a user may provide signals to computing system 414; computing system 414 may interpret these signals as indicating a particular user request or information. User input device 422 may include any or all of the following: a keyboard, a touchpad, a touch screen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, a sensor (e.g., a motion sensor, an eye tracking sensor, etc.), etc.
[0069] User output device 424 may include any device via which computing system 414 can provide information to a user. For example, user output device 424 may include a display to display images generated by or delivered to computing system 414. The display may combine various image generation technologies with supporting electronic devices (e.g., digital-to-analog converters or analog-to-digital converters, or signal processors, etc.), such as liquid crystal displays (LCDs), light-emitting diodes (LEDs) including organic light-emitting diodes (OLEDs), projection systems, or cathode ray tubes (CRTs), etc. Devices that act as both input devices and output devices, such as touch screens, may be used. Output device 424 may be provided in addition to or in place of a display. Examples include indicator lights, speakers, tactile "display" devices, printers, and the like.
[0070] Some embodiments include storing computer program instructions in electronic components such as microprocessors, storage devices, and memory in computer-readable storage media (e.g., non-transient computer-readable media). Multiple features in each feature described in this specification may be implemented as a process, which is specified as a program instruction set encoded on a computer-readable storage medium. These program instructions, when executed by one or more processors, cause the processor to perform the various operations indicated in these program instructions. Examples of program instructions or computer codes include machine code and files, which are generated, for example, by a compiler, and the file includes high-level code executed by a computer, electronic component, or microprocessor using an interpreter. Through suitable programming, processor 416 can provide various functions for computing system 414, which are included in any of the functions described herein as being performed by a server or client, or other functions associated with a message management service.
[0071] It will be understood that computing system 414 is illustrative and various variations and modifications may be made. The computer system used in conjunction with the present disclosure may have other capabilities not explicitly described herein. In addition, although computing system 414 is described with reference to a specific block, it should be understood that these blocks are defined for the convenience of description and are not intended to imply a specific physical arrangement of each component part. For example, different blocks may be located in the same facility, in the same server rack, or on the same motherboard. In addition, these blocks do not necessarily correspond to physically different components. Each block may be configured to perform various operations (e.g., by programming a processor or providing a suitable control circuit), and various blocks may or may not be reconfigurable, depending on how to obtain the initial configuration. Each embodiment of the present disclosure may be implemented in a variety of devices, including electronic devices implemented using any combination of circuits and software.
[0072] Systems and methods for holographic communications
[0073] Reference now Figure 5 , Figure 5 An example diagram 500 of a holographic call or communication session through a head wearable device (HWD) 606 according to an example embodiment of the present disclosure is depicted. A holographic call or communication may be or may include a service provided by an XR / AR / VR / MR device such as smart glasses or other HWD (e.g., HWD 606). The service may provide a user with the ability to communicate with other people who are represented in a three-dimensional (3D) graphic overlaid on a physical / spatial environment visible to the user via a display (e.g., a display of AR glasses and / or a VR head-mounted viewer (headset)). In various embodiments, the position and orientation of other users or objects displayed to the user may be determined by one or more servers and / or by smart glasses, HWD, or other devices communicatively coupled to smart glasses, HWD.
[0074] In order to provide holographic communication, various imagers and / or microphones can collect audio / video (A / V) data of the user (e.g., from various angles, directions, perspectives, positions). The imager and / or microphone can transmit A / V data to the server (e.g., via a smartphone or other user device shared / connected via a network). For example, a user device receiving A / V data can compress the A / V data and can send the compressed A / V data to the server. Upon receiving the compressed A / V data, the server can send the compressed A / V data to another device (e.g., another smartphone or user device associated with another user of the holographic communication session). The other device can receive and reconstruct the media for rendering on the HWD. For example, the other device can send the rendered media to the HWD in various formats for rendering to the other user. In the case where the holographic communication session includes multiple users (e.g., three or more users), the server can collect and process the A / V data of each user.
[0075] In the case where a holographic communication session includes multiple users, the server may apply various considerations as part of processing the A / V data. For example, the size representation of the video data of a particular user or object should be scaled to match the background and the size of the other users or objects. Similarly, the audio data may be spatially reconstructed / remapped to match the location of the corresponding user on the display. Additionally, in the case where multiple users participate in a holographic communication session, the server may virtually position each user at a fixed location relative to the other users.
[0076] Reference now Figure 6 , Figure 6 A block diagram of a system 600 for holographic communication according to an example embodiment of the present disclosure is depicted. The system 600 may include a plurality of client systems 602 communicatively coupled to one or more servers 604. The client systems 602 may include corresponding head wearable devices (HWDs) 606, such as virtual reality (VR) head mounted viewers, augmented reality (AR) smart glasses, or other devices. The HWD 606 may be similar to the HWD 250 described above (or may include hardware / software / components similar to the HWD 250 described above). The HWD 606 may be configured to display or enhance graphics on an environment, which may include a physical (or real world) environment in an embodiment where the HWD 606 is an AR HWD 606, or a virtual environment in an embodiment where the HWD 606 is a VR HWD 606. The client system 602 may include an imaging system 608. The imaging system 608 may be configured to at least capture video data of a user corresponding to the client system 602. Reference Figure 7The user end system 602 may include a user device 610. The user device 610 may be or may include a device similar to the one described above with reference to Figure 1 1. The hardware of the UE 120 described herein. For example, the user device 610 may include a smart phone, a mobile device, a tablet computer, a laptop computer, or other user equipment. Although shown as separate devices, in various embodiments, two or more of the devices of each user end system 602 may be combined into a single device. For example, the imaging system 608 may be a component of the hardware of the user device 610.
[0077] As described in more detail below, and in accordance with various embodiments, each of the user systems 602 can establish a holographic communication session with the server 604. The imaging system 508 of each user end system 602 can be configured to collect audio / video (A / V) data of the corresponding user, and can send the A / V data to the user device 610 (e.g., via a local connection or link). In some embodiments, the imaging system 608 can be configured to collect scale data indicating the size, proportion, size, or scale of the user or object of the A / V data. The imaging system 608 can be configured to send the scale data to the user device 610. In addition, in various embodiments, the HWD 606 can be configured to collect direction data indicating the gaze of the corresponding user, and the HWD 606 can be configured to send the direction data to the user device 610. The user device 610 can be configured to transmit, transfer, send, or otherwise provide the A / V data, scale data, and / or direction data to the server 604. The server 604 may be configured to receive A / V data and scale data and orientation data from each of the client systems 602. The server 604 may be configured to manage the holographic communication session by controlling various aspects of the A / V data according to the scale data and orientation data, and may transmit modified A / V data corresponding to one respective user to the client systems of other users for rendering (e.g., via the respective HWD 606).
[0078] The one or more servers 604 may include one or more processors 612. The processor 612 may be similar to the one described above with reference to Figure 1 and Figure 2 The processors 114, 124, 230, 270 described above, and / or Figure 4 The processing unit 416 described above. One or more servers 604 may include a memory 614. The memory 614 may be similar to the above reference Figure 1 The memories 116 and 126 described above, and / or the ... Figure 4The one or more servers 604 may include one or more processing engines 616. The one or more processing engines 616 may be or may include any device, component, element, or hardware designed or configured to perform various functions or tasks associated with the one or more servers 604. For example, as described in more detail below, the one or more processing engines 616 may be configured to perform various functions related to establishing and managing holographic communication sessions for multiple user-end systems 602. Note that the various one or more processing engines 616 described herein may be subdivided into a variety of one or more additional processing engines 616, and additionally or alternatively, the various one or more processing engines 616 described herein may be combined into a single processing engine 616.
[0079] The one or more servers 604 may include a session management engine 618. The session management engine 618 may be configured to establish and / or maintain holographic communication sessions for the plurality of client systems 602. In some embodiments, the session management engine 618 may be configured to establish a session in response to receiving a request to establish a session from each of the plurality of user devices 610. For example, a first user of a first client system 602(2) may access an application or resource on a user device 610(1) to initiate a holographic communication session with other user devices 610(2) to 610(N) (e.g., by dialing a number or user name or other user identifier associated with the user device 610(2) to 610(N)). The user device 610(1) may send a request to initiate a holographic session along with the identifiers of the other user devices 610(2) to 610(N) to the one or more servers 604. The session management engine 618 may be configured to receive the request and may establish a session based on the request. The session management engine 618 may be configured to send or forward a request to join the session to the other user devices 610(2)-610(N) using the identifiers of the other user devices 610(2)-610(N).
[0080] refer to Figure 6 and Figure 7 The session management engine 618 can be configured to maintain a location map 620 for each session, the location map including an index 622 associated with each user end system 602 included in the session. Specifically, Figure 7 A graphical representation 700 of a virtualization graph 620 is depicted according to an example implementation of the present disclosure. The location graph 620 may include a table including the location and relative position of each of the users associated with the respective client systems 602, such as Table 1 below.
[0081]
[0082] Table 1 Positioning map for holographic communication session
[0083] In Table 1 above, each row in the table may be an index 622 corresponding to a particular user device 602. Each index may include a location corresponding to the location of the user within the graphical representation 700. For example, the user corresponding to the user device 610(1) may be located at location A, the user corresponding to the user device 610(2) may be located at location B, and so on. Each user end system 602 may be randomly assigned to a location in the virtualization graph based on a first-come, first-served basis, according to various user-defined rules, and the like. Although illustrated as a graphical representation of a round table type, in various embodiments, the session management engine 618 may be configured to receive a type or setting of each user end system 602. For example, one user of the first user end system 602(1) may be seated at a round table, while another user of the second user end system 602(2) may be seated at a rectangular table. The session management engine 618 may be configured to receive the physical settings of the respective users (e.g., from the first user device 610(1) and the second user device 610(2)) as part of joining a session. The session management engine 618 may be configured to merge such information into the index of the respective users.
[0084] As shown in Table 1, each index 622 may include data indicating neighboring devices and / or neighboring users within the positioning map 620. Thus, although various users may have different physical settings, they may "virtually sit next to" or be positioned next to the same user in a session. Figure 7 In the example shown and referring to Table 1, the index of the first user may include data identifying devices of other user-end systems 602 to the right and left of the first user, such as a second user (or a user associated with the second user-end system 602(2)) located at position B and an Nth user (e.g., in the case where eight users are in a holographic communication session, the Nth user may be located at position H). Similarly, the index of the second user (e.g., user-end system 602(2)) may include data identifying the first user (e.g., user device 610(1) associated with the first user of the first user-end system 602(1)) and data identifying a third user located at a third position C. When a new user enters the holographic communication session, the session management engine 618 may be configured to update the session positioning map 620 and may add an index 622 for the new user based on the updated positioning map 620.
[0085] Return to reference Figure 6, one or more servers 604 may include a session data receiving engine 624. The session data receiving engine 624 may be configured to ingest, identify, manage, or otherwise receive session data from each of the plurality of client systems 620. The session data receiving engine 624 may be configured to maintain or manage data flows together with the session data sending engine 638 to maintain data flows between the client systems 620 to provide a holographic communication session.
[0086] Reference now Figure 6 and Figure 8 , the client system 602 may include various devices for collecting session data to send to one or more servers 604. Specifically, Figure 8 Various views of an imaging system 608 are shown according to example embodiments of the present disclosure. Although shown as a separate imaging system 608, in various embodiments, each of the components of the imaging system 608 may be incorporated into or included in the user device 610. In other words, the imaging system 608 may be an imaging system of the user device 610.
[0087] The imaging system 608 can be configured to capture audio / video (A / V) data of a user of the client system 602. In some embodiments, the imaging system 608 can be configured to capture spatial audio via two or more microphones 800, and capture three-dimensional (3D) video via various laser emitters 802, color or image (e.g., red-green-blue (RGB)) sensors 804, one or more depth sensors 806, and the like. The two or more microphones 800 can form a stereo audio capture system. Similarly, although in various embodiments the imaging system 608 is shown as using a laser emitter 802, one or more color or image sensors 804, and one or more depth sensors 806 to capture 3D video, the imaging system 608 can include two or more cameras that are arranged to form a stereo video capture system configured to capture 3D video. Figure 8 As shown, imaging system 608 may include a depth field of view (FOV), image sensor FOV, in both a lateral (or horizontal) direction and a longitudinal (or vertical) direction.
[0088] The imaging system 608 may be configured to communicate, transmit, send, or otherwise provide data corresponding to the FOV of the imaging system 608 to the user device 610. In some embodiments, the imaging system 608 may be configured to provide data to the user device 610 as part of establishing or otherwise registering a session (e.g., as part of negotiating a local link or connection between the imaging system 608 and the user device 610). The imaging system 608 may be configured to provide data in the form of a range or number of bits or pixels in a length direction, a width direction, and / or a depth direction. The imaging system 608 may be configured to send data about microphones, such as microphone type and directions of stereo microphones (e.g., such as left and right directions).
[0089] Reference now Figure 8 and Fig. 9 , the imaging system 608 may be configured to transmit, send, or otherwise provide the A / V data captured by the imaging system 608 to the user device 610. Specifically, Fig. 9 An example image corresponding to video data that can be collected by the imaging system 608 according to an example embodiment of the present disclosure is shown. The imaging system 608 can be configured to collect video data in the A / V data as a point cloud, a mesh, RGB-depth (RGB-D), etc. The imaging system 608 can be configured to collect, detect, or otherwise determine the height and width of the user or object represented by the video data. In some embodiments, the imaging system 608 can be configured to estimate the height and width of the user or object based on the range or distance from the imaging system 608 to the object or user (e.g., various points on the user detected by the imaging system 608), and the FOV data and the percentage or ratio of the object / user relative to the FOV. The imaging system 608 can be configured to calculate a value (e.g., a 12-bit unsigned integer) for representing the height or width of the user (e.g., from 0 millimeters (mm) to 4095mm). The imaging system 608 can be configured to transmit, send, or otherwise provide values representing the height and width (e.g., collectively referred to as scale data) to the user device 610.
[0090] In some embodiments, the imaging system 608 may be configured to calculate the distance between each point in the point cloud and the vertical plane including the depth sensor based on various intrinsic and extrinsic characteristics of the imaging system 608. For example, the imaging system 608 may be configured to calculate or otherwise determine the rotation and translation from the 3D physical world coordinate system to the imager's coordinate system. The imaging system 608 may be configured to determine an intrinsic matrix K of the imaging system based on various parameters of the imaging system 608. The imaging system 608 may be configured to determine the intrinsic matrix K as
[0091]
[0092] The variables or parameters used to calculate the intrinsic matrix K are shown in Table 2, and these variables or parameters used to calculate the intrinsic matrix K may be determined by the operating system of the imaging system 608 or provided in other ways.
[0093] parameter unit definition Precautions fx Floating point X-axis focal length (pixels) fy Floating point Y axis focal length (pixels) xx Floating point X-axis principal point (pixels) cy Floating point Y-axis principal point (pixels) s Floating point Skewness coefficient Zero if the image axes are vertical
[0094] Table 2 Parameters used to determine the intrinsic matrix of the imaging system
[0095] The imaging system 608 may be configured to determine or generate a point cloud from a pair of RGB-D frames by reversing the transformation using the intrinsic matrix K. In various embodiments, the imaging system 608 may be configured to determine the size of an object or person in the video data using one or more applications or resources of the imaging system 608 that provide a bounding box or bounding cube around the object to show measurements (e.g., in length, width, and height). The imaging system 608 may be configured to set a minimum size for the bounding box or bounding cube so that the bounding box fits tightly around the object or user, thereby providing a more accurate measurement of the size of the object or user.
[0096] The imaging system 608 may be configured to send scale data to the user device 610 in a real-time transport control protocol (RTCP) packet. In some embodiments, the imaging system 608 may be configured to send scale data periodically and / or on demand. The RTCP packet may include a packet type, an object identifier, a width, and a height. For example, the RTCP packet may have a format that includes four bits for a packet type (e.g., indicating that the RTCP includes size information for a person or object), four bits for an object identifier (e.g., indicating or identifying a person or object in the video data), and 12 bits for both the width and height of the person or object (e.g., for both width and height, the range is, for example, 0 to 4095).
[0097] HWD 606 can be configured to transmit, send or otherwise provide direction data to user device 610. Direction data can be or can include the position, orientation, direction or other direction data indicating the gaze of the user wearer. In some embodiments, HWD 606 can be configured to provide direction data as an 8-bit signed integer representing the direction of each axis (e.g., in the X-axis or transverse axis, and in the Y-axis or longitudinal axis). HWD 606 can be configured to measure direction data based on a relative position relative to true north, based on a relative position relative to a fixed device or position (e.g., imaging system 608), etc. For example, HWD 606 can include an accelerometer, gyroscope or other motion sensor configured to measure the position or motion of HWD 608 (e.g., relative to the axis or plane of HWD 606). HWD 606 can be configured to determine direction data based on or according to the measurement result of the motion sensor from HWD 606. HWD 606 can be configured to send direction data to user device 610.
[0098] HWD 606 may be configured to send directional data to a user device in an RTCP packet. Similar to an RTCP packet including scale data, HWD 606 may be configured to send an RTCP packet with directional data on demand and / or periodically. The RTCP packet may include a packet type, an HWD identifier, an X direction, a Y direction, and / or a Z direction. For example, the RTCP packet may have a format that includes four bits of a packet type (e.g., indicating that the RTCP includes size information of a person or object), four bits of an HWD identifier (e.g., indicating or identifying the HWD 606 used to collect directional data), and / or an 8-bit signed integer value (e.g., ranging between 0 mm and 255 mm) determined for each of the X direction, the Y direction, and the Z direction.
[0099] Reference now Fig.10 , Fig.10 A schematic diagram of an end-user system 602 in communication with a server 604 is depicted according to an example implementation of the present disclosure. Fig.10As shown, HWD 606 can maintain a wireless local area network (WLAN) (e.g., Wi-Fi) connection with user device 610, and user device 610 can maintain a cellular connection with server 604 (although shown as a 5G connection, any type or form of cellular connection may be suitable). In this example, imaging system 606 can be local or native to user device 610. HWD 606 can be configured to transmit or send direction data to user device 610 (e.g., via a WLAN connection). User device 610 can be configured to collect A / V data and scale data via a camera and microphone, and send the A / V data and scale data to server 604 (e.g., via a cellular connection). As described in more detail below, the server 604 can be configured to send other A / V data (e.g., A / V data of other users in the holographic communication session) back to the user device 610, which can send the A / V data to the HWD 606 via the WLAN connection for rendering via one or more speakers and displays.
[0100] Return to reference Figure 6 , the session data receiving engine 624 of the one or more servers 604 can be configured to receive A / V data 626 from the corresponding communication of each of the multiple client systems 602 in the holographic communication session. The session data receiving engine 624 can be configured to also receive scale data 628 (e.g., from the imaging system 608 of the corresponding client system 602) and direction data 630 (e.g., from the HWD 606 of the corresponding client system 602). As described in more detail below, the one or more servers 604 can be configured to use the scale data 628 to generate, determine, derive, or otherwise provide modified video data based on the scale data 628, and can use the direction data 630 to select a bit rate for sending the modified A / V data to the client system 602 for rendering.
[0101] The one or more servers 604 may include a session data processing engine 632 that includes a scaler 634 and a field of view (FOV) determiner 636. As a brief overview, the scaler 634 may be configured to change, adjust, normalize, update, or otherwise modify the proportions, relative sizes, or scales of objects or users depicted in the video of the A / V data based on the scale data 628 received from the imaging system 608 of the corresponding client system 602. The FOV determiner 636 may be configured to identify, detect, derive, compute, calculate, or otherwise determine the field of view of each of the corresponding users based on the direction data 630 received from the HWD 606 of the corresponding client system 602.
[0102] Reference now Figure 6 as well as Fig.11A and Fig. 11B , the scaler 634 may be configured to modify the scale of an object or user depicted in the video data of the A / V data according to the scale data 628. Specifically, Fig.11A and Fig. 11B 1 shows examples of video data frames from three different client systems before and after scaling modification according to an example embodiment of the present disclosure. Fig.11A and Fig. 11B As shown, the video data may originate from separate client systems 602(1)-602(3) and may include representations of three different users 1102, 1104, 1106. The first user 1102 may be a child having a height of approximately three feet, the second user 1104 may be an adult having a height of approximately six feet, and the third user 1106 may be an adult having a height of approximately five feet. As described above, the client systems 602(1)-602(3) may be configured to provide scale data for each of the users 1102-1106 to one or more servers 604.
[0103] The scaler 634 can be configured to receive video data (e.g., in A / V data 626) and scale data 628 from client systems 602(1) to 602(3). In some embodiments, the scaler 634 can be configured to modify the scale of a user or object depicted in the video data based on the scale data 628 from each of the client systems 602. The scaler 634 can be configured to modify the scale of the user or object by increasing and / or decreasing a virtual representation of the user or object in the video data relative to the scale data 628 from other client systems 602 based on the scale data 628 from the respective client system 602. In some embodiments, the scaler 634 can be configured to modify the scale of an object or user depicted in video data from the same client system 602 based on the scale data 628 from the first client system 602 and the scale data 628 from the other client systems 628. The scaler 634 can be configured to modify the scale (e.g., size, dimensions) of an object or user to normalize the scale of the object relative to other objects. For example, assuming that the scale data 628 from the first user-end system 602 is 20% larger than the scale data 628 from the second user-end system 602, the scaler 630 can be configured to modify the scale of the object or user depicted in the video data from the first user-end system 602 (by reducing the scale of the object (e.g., reducing the scale of the object by 20%)), and / or modify the scale of the object depicted in the video data from the second user-end system 602 (by increasing the scale of the object (e.g., increasing the scale of the object by 20%)). Therefore, in response to modifying the scale of the object, each of the users or objects represented in the modified video data can have a normalized scale (e.g., across the A / V data) to display a substantially accurate relative proportion.
[0104] continue Fig.11AIn the example shown, the scaler 634 may be configured to modify the scale of the first user 1102 based on the scale data 628 from the first client system 602 (e.g., indicating a height of three feet) relative to the scale data 628 from another client system 602 (e.g., indicating a height of six feet for the second user 1104) to reduce the scale of the first user 1102 represented in the video data in the A / V data 626 from the first client system 602. Similarly, the scaler 634 may be configured to modify the scale of the third user 1106 based on the scale data 628 from the third client system 602 (e.g., indicating a height of six feet) relative to the scale data 628 from another client system 602 (e.g., indicating a height of six feet for the second user 1104 and / or indicating a height of three feet for the first user) to increase the scale of the third user 1102 represented in the video data in the A / V data 626 from the third client system 602(3). like Fig. 11B As shown, after the scaler 634 modifies the scale of the users depicted in the corresponding video data of the A / V data 626, the modified video data corresponding to the corresponding users can have an appropriate relative scale (e.g., displaying the first user 1102 as approximately half the height of the second user 1104, displaying the second user 1104 as approximately 20% taller than the third user 1108, and displaying the third user 1108 as approximately 65% taller than the first user 1102).
[0105] Reference again Figure 6 and Figure 7 , the FOV determiner 636 can be configured to receive the directional data 630 (in Figure 7630 (represented as vector 704 in the directional data 630) to detect, calculate, compute, identify, or otherwise determine the FOV 702 of a user corresponding to the user-side system 602. The FOV determiner 636 can be configured to determine the FOV 702 based on the localization map 620 and the directional data 630 maintained by the session management engine 618. More specifically, the FOV determiner 636 can be configured to determine the FOV 702 of a particular user based on the directional data 630 and the position of the user (e.g., the user's face, eyes, or gaze) in the localization map. In some embodiments, the FOV determiner 636 can be configured to identify or determine a viewing range that is applied to the directional data 630 to determine the FOV 702. The viewing range can be a default viewing range or a standard viewing range (e.g., 20° to the left and right and top and bottom of the vector 704 defined by the directional data 630). The viewing range can be specific to the HWD 604 (e.g., and can be provided to one or more servers 604 as part of establishing a holographic communication session). The FOV determiner 636 may be configured to determine the FOV 702 as a viewing range in the X direction and the Y direction applied to the vector 704. The FOV determiner 636 may be configured to determine the FOV 702 for each of the users corresponding to the respective client system 602.
[0106] FOV determiner 636 may be configured to identify, detect, or otherwise determine the location of an object or user relative to FOV 702 (e.g., as reflected in localization map 620). FOV determiner 636 may be configured to use localization map 620 to determine the location of an object or user relative to FOV 702. FOV determiner 636 may be configured to apply FOV 702 to localization map 620 to determine which objects or users are located in the FOV 702 of the respective user. For example, FOV determiner 636 may be configured to project FOV 702 onto localization map 620 to determine which objects or users are located at locations that overlap or intersect with FOV 702. Figure 7 In the example shown, after projecting FOV 702 onto localization map 620, FOV determiner 636 can be configured to determine that the FOV 702 of users at positions D through F intersects or overlaps with the FOV 702 of the user at position A. FOV determiner 636 can be configured to determine the position of each of the users relative to FOV 702 using indexes 622(1) through 622(N).
[0107] refer to Figure 6, one or more servers 604 may include a session data sending engine 638. The session data sending engine 638 may be configured to select a bit rate 640 for sending the A / V data to the client system 602 (e.g., from a plurality of bit rates 640(1) to 640(N)). The session data sending engine 638 may be configured to select a bit rate 640 for compressing the A / V data. The A / V data may be or may include modified A / V data (e.g., after the scaler 634 modifies the scale of the object according to the scale data 628). The session data sending engine 638 may be configured to select a bit rate 640 for sending given A / V data associated with the source client system 602 to the recipient client system 602 based on a position assigned to a user corresponding to the source client system 602 relative to the FOV 702 corresponding to the recipient client system 602. Therefore, the session data sending engine 638 can be configured to compress the same A / V data at different bit rates for different receiving user-end systems 602, based on the position of the source of the A / V data (e.g., the source user-end system 602 that generated the A / V data) relative to the FOV 702 of the receiving user-end system 602 (e.g., by the session management engine 618).
[0108] In some embodiments, the session data transmission engine 638 may be configured to select a bit rate 640 based on whether the location assigned to the user corresponding to the source client system 602 is within the FOV 702 corresponding to the recipient client system 602. In other words, the session data transmission engine 638 may be configured to select a first bit rate 640 for A / V data from the source client system 602 whose location is within the FOV 702 of the recipient client system 602, and to select a second bit rate 640 for A / V data from the source client system 602 whose location is outside the FOV 702 of the recipient client system 602. In this example, the first bit rate 640 may be higher than the second bit rate 640, thereby resulting in higher quality / clarity (e.g., less pixelation) of the video data of the A / V data of the user within the FOV 702. In some embodiments, the session data transmission engine 638 can be configured to select a bit rate 640 for A / V data associated with a source client system 602 whose location is not within the FOV 702 based on the proximity of the location to the FOV 702. For example, the session data transmission engine 638 can be configured to select a bit rate 640 for A / V data associated with the source client system 602 to increase as the proximity of the location corresponding to the source client system 602 is closer to the FOV 702.
[0109] Reference now Fig.12 , Fig.12A flowchart illustrating an example method for updating session conditions of a client system according to an example embodiment of the present disclosure is depicted. The method 1200 may be performed by Figure 6 604). As a brief overview, at step 1202, method 1200 may begin. At step 1204, one or more servers 604 may receive a request to join a session. At step 1206, one or more servers 604 may determine whether a session is ongoing. At step 1208, one or more servers 604 may reconfigure A / V data for other client systems 602. At step 1210, one or more servers 604 may send updates to client systems 602. At step 1212, one or more servers 604 may send updates to requesting client systems 602.
[0110] At step 1202, method 1200 may begin. Method 1200 may begin when one or more servers 604 generate a new session. For example, one or more servers 604 may generate a new session in response to one or more users requesting a new session on their respective client systems 602. One or more servers 604 may establish a session in response to a request from client system 602. Thus, in response to establishing a new session, one or more servers may begin executing steps 1202 to 1212 in method 1200. One or more servers 604 may be configured to execute method 1200 for each session established by one or more servers 604.
[0111] In step 1204, one or more servers 604 may receive a request to join a session. One or more servers 604 may receive a request to join a session from a user end system 602. The user end system 602 may initiate a request including an identifier or other identification information of a specific session. For example, a user of the user end system 602 may be configured to control the user device 610 to select: a link to join a session invitation, a code or identifier for entering a specific session, etc. The user end system 602 may be configured to send a request including a session identifier to one or more servers 604. The user end system 602 may be configured to send a request to one or more servers 604 using a cellular connection or link between the user end system 602 and the one or more servers 604. The one or more servers 604 may be configured to receive a request from the user end system 602.
[0112] At step 1206, one or more servers 604 may determine whether the session is ongoing. In some embodiments, one or more servers 604 may be configured to determine whether the session is ongoing by performing a session lookup using a session identifier. One or more servers 604 may be configured to determine whether the session is ongoing in response to determining that one or more additional user end systems 602 are currently active in the session or otherwise included in the session. One or more servers 604 may be configured to determine that the session is not ongoing in response to (e.g., in response to performing a lookup using a session identifier) determining that the identifier is not a known identifier and / or that no user is active in the session. At step 1206, in the case where one or more servers 604 (e.g., upon request) determine that the session is not an ongoing session, method 1200 may proceed to step 1212. At step 1206, in the case where one or more servers 604 determine that the session is ongoing, method 1200 may proceed to step 1208.
[0113] In step 1208, one or more servers 604 can reconfigure A / V data for other client systems 602. In some embodiments, one or more servers 604 can reconfigure A / V data received from client systems 602 currently active in the session, and / or reconfigure A / V data received from client systems 602 joining the session. One or more servers 604 can reconfigure A / V data by modifying such bit rate: the bit rate is used to encode A / V services sent from one or more servers 604 to client systems. For example, in the case where HWD606 has a limited display or resolution, as more users participate in the session, one or more servers 604 can reduce or lower the bit rate of three-dimensional (3D) video data, so fewer display pixels can be allocated to represent each user or object. One or more servers 604 can adjust session conditions (e.g., the bit rate used to encode A / V data, the bit rate used to encode A / V data for client systems 602 for decoding at one or more servers 604, etc.).
[0114] At step S1210, one or more servers 604 may send updates to client systems 602. One or more servers 604 may send updates to client systems 602 that are currently in a session. One or more servers 604 may send updates to identify new bit rates that will be used by one or more client systems 602 to encode and / or decode A / V data to / from one or more servers 604. One or more servers 604 may send updates to each of the client systems 602 to set, establish, or otherwise update session conditions for each of the client systems 602. At step 1212, one or more servers 604 may send updates to the requesting client system 602. Similar to step 1210, one or more servers 604 may send updates to client systems 602 that request to join a session. Similarly, (eg, at step 1206 ) in the event that a session is not active or ongoing, one or more servers 604 may send an update to the client system 602 to establish a session with the single client system 602 .
[0115] Reference now Fig.13 , Fig.13 A flow chart illustrating an example method of updating the field of view (FOV) of a device according to an example embodiment of the present disclosure is depicted. The method 1300 may be performed by Figure 6 The process is performed by one or more of the devices, components, or hardware in the system (e.g., the client system 602 and / or one or more servers 604). Although described herein as being performed by the client system 602, similar functions and steps may be performed by the above reference system. Figure 6 and Figure 7 The method 1300 may be performed by one or more servers 604 as described. As a brief overview, at step 1302, the method 1300 may begin. At step 1304, the client system 602 may send a request to join a session. At step 1306, the client system 602 may determine the format and bit rate. At step 1308, the client system 602 may receive the index and identifier. At step 1310, the client system 602 may start the session. At step 1310, the client system 602 may determine whether the FOV has changed. At step 1312, the client system 602 may send an updated FOV.
[0116] At step 1302, method 1300 may begin. Method 1300 may begin when user device 606 is turned on. Method 1300 may begin when user device 606 establishes a connection (e.g., a local connection) with other components or elements of client system 602. Method 1300 may begin when user device 606 opens or otherwise launches an application for a holographic communication session.
[0117] At step 1304, the client system 602 may send a request to join the session. Step 1304 may be similar to the above referenced Fig.12 Step 1204 described. Client system 602 may send a request including an identifier or other information related to the session. For example, client system 602 may use an application or resource on user device 606 to send the request and include an identifier for the session in the request (e.g., by selecting a link for the session or typing in a code for the session, as some possibilities). Client system 602 may be configured to send the request to one or more servers 604.
[0118] At step 1306, client system 602 may determine a format and a bit rate. Client system 602 may identify various formatting information corresponding to the A / V data captured by client system 602. For example, client system 602 may identify the formatting information from an application, from one or more servers 604, from local information stored on an operating system of one or more components of client system 602, etc. Client system 602 may identify a media format of the 3D video, such as a codec type, a resolution of the 2D video including the projected 3D video element, a maximum number of points to be captured and compressed, etc.
[0119] At step 1308, the client system 602 may receive an index and an identifier. The client system 602 may receive an index and a device identifier for the client system 602 from one or more servers 604. As described above, one or more servers 604 may maintain an index for each client system 602 in each client system in the session, wherein each index includes (in addition to other information) a device identifier corresponding to the client system 602. One or more servers 604 may send an index for each client system 602 in each client system 602 in the session to a new client system 602 to build or otherwise maintain a positioning map. Thus, in the case where the current session is active, one or more servers 604 may assign a unique identifier and a position index to the client system 602 to join the current session, and may share the unique identifier and position index with other client systems 602. The position index may be or may include a real number or an integer indicating the order of 3D objects in a clockwise or counterclockwise manner. The first object and the last object may be assigned a special type of object identifier or location index (eg, object of the first ID (object_id_first), object of the last ID (object_id_last)). The last object may be assigned the largest location index (location_index) value.
[0120] At step 1310, the client system 602 may start a session. The client system 602 may send A / V data of a corresponding user of the client system 602 to one or more servers 604, and one or more servers 604 may send A / V data of other users of other client systems 602 back to the client system 602.
[0121] At step 1310, the client system 602 may determine whether the FOV has changed. The client system 602 may determine whether the FOV has changed based on or according to sensor data of the client system 602. For example, the client system 602 may determine that the FOV has changed in response to detecting motion from a motion sensor such as a gyroscope and / or an accelerometer. The client system 602 may be configured to determine that the FOV has changed in response to data from the HWD 606 of the client system 602. In other words, the client system 602 may detect the FOV (including changes in the FOV) based on data from one or more sensors of the HWD 606.
[0122] In step 1312, the user-side system 602 may send the updated FOV. In some embodiments, the user-side system 602 may send data corresponding to the updated FOV to one or more servers 604. The user-side system 602 may send data corresponding to an object or user (e.g., a virtual representation of an object or user) currently within the updated FOV. In this regard, the user-side system 602 may send a list or other identifier of devices that have a location index within the FOV. In some embodiments, the user-side system 602 may send data corresponding to the coordinates of the FOV to one or more servers 604. One or more servers 604 may receive the data corresponding to the coordinates of the FOV and may determine users corresponding to the user-side systems 602 that are assigned to locations within the FOV of the user-side system 602.
[0123] The method 1300 may loop between steps 1312 and 1314 until the corresponding client system 602 terminates the session (eg, by logging out of the session, shutting down the HWD 606 and / or other components of the client system 602, etc.) When the client system 602 terminates the session, the method 1300 may end.
[0124] Reference now Fig.14 , Fig.14 A flowchart illustrating an example method for managing bit rates for objects in a communication session according to an example implementation of the present disclosure is depicted. The method 1400 may be performed by Figure 6, or one or more of the devices, components, or hardware in (e.g., one or more servers 604). As a brief overview, at step 1402, method 1400 can begin. At step 1404, one or more servers 604 can receive a request to join a session. At step 1406, one or more servers 604 can determine the format and bit rate. At step 1408, one or more servers 604 can update the index and positioning map. At step 1410, one or more servers 604 can continue the session. At step 1412, one or more servers 604 can determine whether a new field of view (FOV) has been received. At step 1414, one or more servers 504 can adjust the bit rate. At step 1416, one or more servers 604 can determine whether a request to terminate the session has been received.
[0125] At step 1402, method 1400 may begin. Similar to step 1202, method 1400 may begin when one or more servers 604 generate a new session. For example, one or more servers 604 may generate a new session in response to one or more users requesting a new session on their respective client systems 602. One or more servers 604 may establish a session in response to a request from a client system 602. One or more servers 604 may be configured to perform method 1400 for each session established by one or more servers 604.
[0126] At step 1404, one or more servers 604 may receive a request to join the session. Step 1404 may be similar to step 1204 described above. At step 1406, one or more servers 604 may determine a format and a bit rate. One or more servers 604 may determine a media format and a bit rate for sending and receiving (e.g., sending to and receiving from the client system 602) the A / V data. One or more servers 604 may determine a media format including a codec type, a resolution of a 2D video, a maximum number of points collected and compressed at the client system, an encoding bit rate for sending data from the client system 602 to the one or more servers 604, a decoding bit rate for sending data from the one or more servers 604 to the client system 602, and the like.
[0127] In step 1408, the one or more servers 604 may update the index and the location map. The one or more servers 604 may update the index and the location map to add the client system 602 that generated the request in step 1404. The one or more servers 604 may update the index to include the new device identifier at the location index that is not currently used by other client systems 602. For example, in the case where a client system 602 is added to the current session, the one or more servers 604 may assign a unique identifier and location index that may be shared by the one or more servers 604 or otherwise sent to other client systems 602. In the case where a new client system is inserted between two location indexes, the one or more servers 604 may assign the client system a location index that is an average between new adjacent location indexes (e.g., for adjacent client systems 604). In addition, in the case where a new object or user-side system is inserted between the first object and the last object, the new object or user-side system 602 may be assigned a value greater than the location index of the previous last object (e.g., by one or more servers 604), and the user-side system 602 may be the new last object. In addition, in the case where the user-side system 602 that was previously the first object (or the last object) leaves the conference session, the one or more servers 604 may assign the next object (or the previous object) as the first object (or the last object) by updating the device identifier and the location index.
[0128] At step 1410, one or more servers 604 may continue the session.
[0129] At step 1412, one or more servers 604 may determine whether a new field of view (FOV) has been received. In some embodiments, one or more servers 604 may determine whether a new FOV has been received from one of the user-end systems 602. One or more servers 604 may receive data indicating a new FOV in response to execution of a step (e.g., step 1314) in method 1300. One or more servers 604 identify or determine a new FOV based on data received from the user-end system. For example, the data may be or may include directional data. The directional data may be a gaze direction of a user corresponding to the user-end system 602, a set of device identifiers included in the FOV of the user-end system 602, and the like. One or more servers 604 may identify a user-end system 602 corresponding to a position within the FOV of the user-end system 602 (e.g., with an updated FOV).
[0130] At step 1414, the one or more servers 504 may adjust the bit rate. The one or more servers 604 may adjust the bit rate used to compress the A / V data sent from the one or more servers 504 to a client system 602. The one or more servers 604 may adjust the bit rate used to compress the A / V data based on which client systems 602 correspond to positions within the FOV of the client system 602 and which client systems 602 correspond to positions outside the FOV of the client system 602. The one or more servers 604 may adjust the bit rate to compress the A / V data from the client system 602 corresponding to the position within the FOV at a higher bit rate than the A / V data from the client system 602 corresponding to the position outside the FOV. The one or more servers 604 may select the bit rate used to compress the A / V data from the client system 602 corresponding to the position outside the FOV to be gradually larger as the position approaches the FOV (e.g., closer to the bit rate used for the client system 602 corresponding to the position within the FOV). In this regard, as the location corresponding to the location of the source client system 602 of the A / V data approaches the FOV, the bit rate used to compress the A / V data can be gradually increased. The one or more servers 604 can compress the A / V data for transmission to the corresponding client system 602 according to the bit rate selected for the A / V data (e.g., the bit rate selected based on the location of the source client system 602 relative to the FOV of the user of the corresponding client system 602).
[0131] At step 1416, one or more servers 604 may determine whether a request to terminate the session has been received. Thus, the method 1400 may loop between steps 1412 and 1414 until a request to terminate the session is received from the end-user system 602. In the event that a request is received at step 1416, the method 1400 may loop back to step 1408 to update the index and location map. The method 1400 may continue with steps 1408 to 1416 until no more end-user systems 602 are joined to the session / in the session.
[0132] Reference now Fig.15 , Fig.15 Depicted is a flowchart illustrating an example method 1500 for sending information for holographic communication according to an example embodiment of the present disclosure. The method 1500 may be as described above with reference to Figures 1 to 14 As a brief overview, at step 1502, one or more servers may maintain a session. At step 1504, one or more servers may receive audio / video (A / V) data and scale data. At step 1506, one or more servers may modify the scale. At step 1508, one or more servers may send the modified video data.
[0133] At step 1502, one or more servers may maintain a session. In some embodiments, the one or more servers may maintain a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. In some embodiments, method 1500 may establish multiple sessions (e.g., multiple sessions including a first session and a second session) before step 1502. The one or more servers may establish a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users. The one or more servers may establish the first session and the second session in response to or based on a request for a conference call between the first device and the one or more second devices. The one or more servers may establish the first session and the one or more second sessions in response to the first device and the one or more second devices joining a session (or conference call) between the respective devices.
[0134] In step 1504, one or more servers may receive audio / video (A / V) data and scale data. In some embodiments, one or more servers may receive audio / video (A / V) data of a first user of a first device and scale data of the first user via a first session. The first device may collect, detect or otherwise identify A / V data and scale data. For example, the first device may receive A / V data and scale data from an imaging system communicatively coupled to the first device. The first device may forward, transmit, send or otherwise provide A / V data and scale data to one or more servers. One or more servers may receive A / V data of the first user at a period or frequency set or established as part of maintaining a holographic communication session between one or more devices. One or more servers may receive scale data at a second frequency. For example, one or more servers may receive scale data at various intervals, which may be every 10 seconds, 30 seconds, 1 minute, etc. Therefore, one or more servers may receive scale data separate from the A / V data of the first user.
[0135] In some embodiments, one or more servers may receive the second A / V data and the second scale data of the second user. One or more servers may receive the second A / V data and the second scale data through one or more second sessions maintained with one or more second devices (e.g., in step 1502). Similar to the first device, one or more second devices may send the second A / V data and the second scale data received from the HWD communicatively coupled to the one or more second devices to one or more servers through the second session. One or more second devices may send the second A / V data and the second scale data at an interval similar to the first A / V data and the scale data (e.g., from the first device). In response to the second device sending the second A / V data and the second scale data, one or more servers may receive the A / V data and the scale data.
[0136] In step 1506, one or more servers may modify the scale. In some embodiments, one or more servers may modify the scale of the first user (or object) represented in the video data of the A / V data according to the scale data. One or more servers may modify the scale of the first user represented in the video data according to the scale data from the corresponding first device. One or more servers may modify the scale of the first user according to the scale data from each device (e.g., the first device and the second device). One or more servers may modify the scale of the first user to normalize the scale of the first user from the first A / V data relative to the scale of one or more second users represented in the video data of the second A / V data. Therefore, in some embodiments, one or more servers may modify the scale of the second user represented in the second video data of the second A / V data according to the second scale data (and the first scale data). In some embodiments, one or more servers may modify the scale of the user represented in the first video data according to the first scale data and the second scale data. Similarly, one or more servers may modify the scale of one or more second users represented in the second video data according to the first scale data and the second scale data. The one or more servers may modify the scale of a first user represented in the first video data based on the first scale data and the second scale data to match the scale of one or more second users represented in the second video data. In this regard, the scale of users across A / V data from user devices may be normalized by the one or more servers based on the scale data from the corresponding user devices.
[0137] In step 1508, the one or more servers may send the modified video data. In some embodiments, the one or more servers may send the modified A / V data of the first user to one or more second devices (e.g., via one or more second sessions) for rendering to the one or more second users (e.g., after modifying the scale of the video data). The one or more servers may send the modified A / V data to each of the second devices of the second users to render the modified visual data to the second users (e.g., after adjusting the scale of the first user in the visual data). In some embodiments, the one or more servers may also send the modified second A / V data of the one or more second users to the first device via the first session for rendering to the first user of the first device.
[0138] Reference now Fig.16 , Fig.16 Depicted is a flow chart showing an example method 1600 for improving field of view according to an example embodiment of the present disclosure. The method 1600 may be as described above with reference to Figures 1 to 14 As a brief overview, at step 1602, one or more servers may maintain a positioning map. At step 1604, one or more servers may receive audio / video data. At step 1606, one or more servers may receive direction data. At step 1608, one or more servers may send rendering data.
[0139] In step 1602, one or more servers may maintain a positioning map. In some embodiments, one or more servers may maintain the relative position of each user in a plurality of users in the positioning map. One or more servers may maintain the relative position of the corresponding device assigned to each user in a plurality of users in the positioning map. In some embodiments, one or more servers may assign relative positions to each user device of the corresponding user included in the session. One or more servers may maintain a positioning map, which includes a device identifier and a corresponding position assigned to each corresponding device. In some embodiments, one or more servers may maintain the positioning map as a position index for each user. The position index in each index may include or may otherwise indicate the position assigned to the user relative to at least some of the plurality of users. In some embodiments, the position index may include or may otherwise indicate the position assigned to the corresponding user relative to each user in a plurality of users. In some embodiments, the position index may include or may otherwise indicate the position assigned to the corresponding user relative to the nearest neighboring user (e.g., the user assigned to the position) of the user.
[0140] In step 1604, one or more servers may receive audio / video data. In some embodiments, one or more servers may receive audio / video (A / V) data of a first user from a first device. One or more servers may receive the user's A / V data through a session maintained between the server and the first device. The first device may detect, determine, identify, or otherwise generate A / V data in response to the acquisition of A / V data by an imaging system communicatively coupled to the first device. The first device may send the A / V data to the server through the session. In step 1606, one or more servers may receive direction data. In some embodiments, one or more servers may receive direction data from a second device, the direction data indicating the gaze of a second user of a plurality of users. The direction data may include a vector or coordinate of the gaze, an identifier of a user device associated with a place or position within a field of view (FOV) of the second user, and the like. The second device may detect, determine, or otherwise identify the direction data based on data from one or more sensors of a head wearable device (HWD) communicatively coupled to the second device. The second device may send the direction data to one or more servers via a second session. One or more servers may determine or identify one or more devices associated with a position within the FOV of the second user. In some embodiments, the one or more servers may determine the FOV based on the directional data (e.g., by applying the viewing range to the vector or coordinates of the gaze), and determine which devices are assigned to positions within the FOV (e.g., based on a localization map). In some embodiments, the one or more servers may determine which devices are assigned to positions within the FOV based on a device identifier included as directional data from the second device.
[0141] In some embodiments, one or more servers may select a bit rate from a plurality of bit rates based on the position of the second user relative to the directional data indicating the gaze of the first user. For example, one or more servers may select a bit rate for the A / V data of the source user device based on the position of the FOV assigned to the source user device relative to the user device determined to receive the A / V data. One or more servers may select a bit rate for compressing the A / V data for transmission to the user device. In this regard, for a given source user device, one or more servers may select different bit rates to compress the A / V data from the source user device for transmission to different recipient user devices based on the location or position of the FOV assigned to the source user device relative to the recipient user device.
[0142] In some embodiments, one or more servers may identify a subset of location indexes of a subset of users within a viewing range (or FOV) of a second user based on the location index and direction data of the second user. One or more servers may select a bit rate for rendering data of a subset of users based on the location index associated with the users within the FOV of the second user. In some embodiments, one or more servers may select a bit rate for a subset of users assigned to locations within the viewing range of the second user to be higher than a bit rate for rendering data of other users outside the subset (e.g., outside the viewing range). In some embodiments, one or more servers may select a bit rate for A / V data of other users outside the viewing range based on the proximity of the location index of other users relative to the viewing range. For example, one or more servers may select a bit rate for rendering data of corresponding users of other users to increase as the proximity of the location index of the corresponding user relative to the viewing range decreases.
[0143] In step 1608, one or more servers may send rendering data. In some embodiments, one or more servers may send rendering data to a second device corresponding to the first A / V data. One or more servers may send the rendering data to the second device, and the rendering data is compressed at a bit rate selected according to the direction data and the relative position of the second user in the positioning map. One or more servers may send rendering data in response to compressing the rendering data at the selected bit rate. One or more servers may send rendering data to the second device for decompression and rendering (e.g., via a HWD communicatively coupled to the second device). One or more servers may similarly receive direction data indicating the gaze of other users, receive the A / V data of the second user, and send rendering data corresponding to the second A / V data compressed at a second bit rate. In this regard, one or more servers may compress the A / V data sent to the user device at different bit rates based on whether the A / V data corresponds to a position within the FOV of the user device receiving the compressed A / V data.
[0144] Now that some illustrative embodiments have been described, it is apparent that the foregoing, which has been presented as examples, is illustrative and not restrictive. Specifically, although many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and these elements may be combined in other ways to achieve the same purpose. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from other embodiments or similar roles in the embodiments.
[0145] The hardware and multiple data processing components for implementing the various processes, operations, illustrative logic, logic blocks, modules and circuits described in conjunction with the multiple embodiments disclosed herein can be implemented or executed using the following components designed to perform the functions described herein: a general-purpose single-chip processor or a general-purpose multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination thereof. A general-purpose processor can be a microprocessor, or can be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. In some embodiments, specific processes and methods can be performed by circuits dedicated to a given function. The memory (e.g., memory, memory unit, storage device, etc.) may include one or more devices (e.g., RAM, ROM, flash memory, hard disk storage device, etc.) for storing data and / or computer code for completing or facilitating the various processes, various layers, and various modules described in the present disclosure. The memory may be or may include volatile memory or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. According to an exemplary embodiment, the memory is communicatively connected to the processor via the processing circuit and includes computer code for executing (e.g., by the processing circuit and / or the processor) one or more processes described herein.
[0146] The present disclosure contemplates methods, systems, and program products on any machine-readable medium for completing various operations. Embodiments of the present disclosure may be implemented using existing computer processors, or implemented by a dedicated computer processor for a suitable system incorporated for this or other purposes, or implemented by a hard-wired system. Embodiments within the scope of the present disclosure include program products, which include machine-readable media for carrying or storing machine-executable instructions or data structures thereon. Such machine-readable media may be any available medium that can be accessed by a general-purpose computer or a special-purpose computer or other machine with a processor. As an example, such machine-readable media may include RAM, ROM, EPROM, EEPROM, or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, or may include any other medium as follows: the other medium may be used to carry or store the desired program code in the form of machine-executable instructions or data structures, and may be accessed by a general-purpose computer or a special-purpose computer or other machine with a processor. The above combination is also included in the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing machine to perform a certain function or a certain group of functions.
[0147] The words and terms used herein are for descriptive purposes and should not be considered limiting. The use of "comprising," "including," "having," "containing," "involving," "characterized by," "characterized by," and variations thereof herein is meant to encompass the items listed thereafter, their equivalents and additional items, and to include alternative embodiments consisting exclusively of the items listed thereafter. In one embodiment, the systems and methods described herein consist of one of the described elements, actions, or components, of various combinations of more than one of the described elements, actions, or components, or of all of the described elements, actions, or components.
[0148] Any reference to an embodiment or element or action of the system and method mentioned herein in the singular may also encompass an embodiment including a plurality of these elements, and any reference to any embodiment or element or action herein in the plural may also encompass an embodiment including only a single element. References in the singular or plural are not intended to limit the systems or methods of the present disclosure, their components, actions or elements to a single configuration or multiple configurations. References to any action or element based on any information, action or element may include an embodiment in which the action or element is based at least in part on any information, action or element.
[0149] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment or example. These terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may be combined inclusively or exclusively with any other embodiment in any manner consistent with the various aspects and embodiments disclosed herein.
[0150] Where reference numerals follow technical features in the drawings, detailed description or any claims, these reference numerals have been included to increase the intelligibility of the drawings, detailed description and claims. Therefore, the presence or absence of these reference numerals will not have any limiting effect on the scope of any claim element.
[0151] The systems and methods described herein may be implemented in other specific forms without departing from the characteristics of the systems and methods described herein. Unless otherwise expressly noted, references to terms of "approximately," "about," "substantially," or other degrees include variations of plus / minus 10% (+ / -10%) from a given measurement, unit, or range. Coupled elements may be electrically, mechanically, or physically coupled to each other, directly or through intermediate elements. Therefore, the scope of the systems and methods described herein is represented by the appended claims rather than the preceding description, and variations within the meaning and scope of the equivalents of the claims are encompassed therein.
[0152] The term "coupled" and variations thereof include two components being directly coupled to each other or indirectly coupled. Such coupling may be constant (e.g., permanent or fixed) or movable (e.g., removable or releasable). Such coupling may be achieved by two components being directly coupled to each other or coupled to each other, wherein the two components are coupled to each other using a separate intermediate component and any additional intermediate components that are coupled to each other, or the two components are coupled to each other using an intermediate component that is integrally formed as a single integral body with one of the two components. If "coupled" or variations thereof are modified by additional terms (e.g., directly coupled), the general definition of "coupled" provided above is modified by the plain language meaning of the additional terms (e.g., "directly coupled" means that the two components are coupled without any separate intermediate components), resulting in a narrower definition than the general definition of "coupled" provided above. Such coupling may be mechanical, electrical, or fluidic.
[0153] References to "or" may be interpreted as inclusive, such that any term described using "or" may mean any of a single, more than one, and all of the multiple terms described. References to "at least one of 'A' and 'B'" may include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with "including" or other public terms may include additional items.
[0154] Without departing substantially from the teachings and advantages of the subject matter disclosed herein, modifications may be made to the elements and actions described, such as variations in size, dimensions, structure, shape, and proportion of the various elements, variations in parameter values, mounting arrangements, use of materials, colors, orientations. For example, an element shown as being integrally formed may be composed of multiple parts or elements, the positions of these elements may be reversed or otherwise changed, and the nature or number or position of discrete elements may be altered or varied. Other substitutions, modifications, changes, and omissions may also be made to the design, operating conditions, and arrangements of the disclosed elements and operations without departing from the scope of the present disclosure.
[0155] References herein to the position of elements (e.g., "top," "bottom," "above," "below") are only used to describe the orientation of the various elements in the drawings. According to other exemplary embodiments, the orientation of the various elements may be different, and such variations are intended to be encompassed by the present disclosure.
Claims
1. A method comprising: maintaining, by one or more servers, a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users; receiving, by the one or more servers via the first session, audio / video (A / V) data of a first user of the first device and scale data of the first user; modifying, by the one or more servers, a scale of the first user represented in the video data of the A / V data based on the scale data; as well as The modified A / V data of the first user is sent by the one or more servers to the one or more second devices via the one or more second sessions for rendering to the one or more second users.
2. The method according to claim 1, further comprising: The first session and the one or more second sessions are established by the one or more servers based on a request for a three-dimensional (3D) communication session between the first device and the one or more second devices.
3. The method according to claim 1 or claim 2, further comprising: receiving, by the one or more servers via the one or more second sessions, second A / V data of the one or more second users and second scaled data of the one or more second users from the one or more second devices; modifying, by the one or more servers, a scale of the one or more second users represented in the second video data of the second A / V data based on the second scale data; as well as The modified second A / V data of the one or more second users is sent by the one or more servers to the first device via the first session for rendering to the first user of the first device.
4. The method according to claim 3, wherein: Modifying the scale of the first video data includes: the one or more servers modifying the scale of the user represented in the first video data based on the first scale data and the second scale data, and wherein, modifying the scale of the one or more second users represented in the second video data includes: the one or more servers modifying the scale of the one or more second users represented in the second video data based on the first scale data and the second scale data.
5. The method according to claim 3 or claim 4, wherein: Modifying the scale includes modifying the scale of the first user represented in the first video data to match the scale of one or more second users represented in the second video data based on the first scale data and the second scale data.
6. A method according to any one of the preceding claims, wherein: The A / V data includes three-dimensional (3D) video data and spatial audio data.
7. The method according to any one of the preceding claims, further comprising: Data indicative of a field of view (FOV) of the first user is received by the one or more servers.
8. The method according to claim 7, wherein: The data indicative of the (FOV) is determined by the first device based on data received from a third device communicatively coupled to the first device.
9. The method according to claim 7 or claim 8, further comprising: maintaining, by the one or more servers, a relative position of each of the first user, the second user, and the third user in a positioning map for the first user of the first device and at least a second user of a second device and a third user of a third device; as well as Determining, by the one or more servers, the FOV of the first user based on the direction data and the relative position of the first user in the positioning map; and preferably The method further comprises: sending, by the one or more servers, second A / V data of a second user to the first device via the first session at a first bit rate based on the data indicating the FOV of the first user; as well as The one or more servers transmit third A / V data of a third user to the first device via the first session at a second bit rate based on the data indicating the FOV of the first user.
10. One or more servers, including: One or more processors configured to: maintaining a first session with a first device of a first user and one or more second sessions with one or more second devices of one or more second users; receiving audio / video (A / V) data of a first user of the first device and scaled data of the first user via the first session; modifying a scale of video data of the first user represented in the A / V data based on the scale data; as well as The modified A / V data of the first user is sent to the one or more second devices via the one or more second sessions for rendering to the one or more second users.
11. The one or more servers of claim 10, wherein: The one or more processors are configured to establish the first session and the one or more second sessions based on a request for a three-dimensional (3D) communication session between the first device and the one or more second devices.
12. One or more servers according to claim 10 or claim 11, wherein: The one or more processors are configured to: receiving, via the one or more second sessions, second A / V data of the one or more second users and second scaled data of the one or more second users from the one or more second devices; modifying a scale of second video data of the second A / V data based on the second scale data; as well as sending the modified second A / V data of the one or more second users to the first device via the first session for rendering to the first user of the first device; and preferably wherein the one or more processors are configured to: modify the scale of the first video data according to the first scale data and the second scale data, and modify the scale of the second video data according to the first scale data and the second scale data; and / or preferably The one or more processors are configured to modify the scale of the first video data according to the first scale data and the second scale data to match the scale of the second video data.
13. One or more servers according to claim 10, claim 11 or claim 12, wherein: The A / V data includes three-dimensional (3D) video data and spatial audio data.
14. One or more servers according to any one of claims 10 to 13, wherein: The one or more processors are configured to: receiving data indicative of a field of view (FOV) of the first user, the data being determined by the first device based on data received from a third device communicatively coupled to the first device; and preferably Wherein, the one or more processors are configured to: maintaining a relative position of each of the first user, the second user, and the third user in a positioning map for the first user of the first device and at least a second user of a second device and a third user of a third device; and determining the field of view of the first user based on the data indicative of the FOV and the relative position of the first user in the positioning map; and / or preferably Wherein, the one or more processors are configured to: transmitting second A / V data of a second user to the first device at a first bit rate via the first session based on the data indicating the FOV of the first user; and Third A / V data of a third user is transmitted to the first device at a second bit rate via the first session based on the data indicating the FOV of the first user.
15. A device comprising: One or more cameras; as well as One or more processors configured to: determining scale data of a user of the first device based on image data collected via the one or more cameras of the first device; receiving audio / video (A / V) data from the user; as well as The A / V data and the scaling data of the user are sent to one or more servers so that the one or more servers modify the A / V data of the user according to the scaling data, and send the modified A / V data of the user to one or more second devices.
Citation Information
Cited By
User-assisted selection of forwarded data in holographic communications
US20260301341A1
