Method and apparatus, system, and storage medium for video conferencing

Abstract

A method, computer program, and computer system for video conferencing are provided. Video data associated with a session of an immersive teleconference is received. A parameter associated with the video data is identified, the parameter specifying overlay data associated with the session of the immersive teleconference. Based on the identified parameter, the video data is displayed with one or more overlays.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 039,336, filed June 15, 2020, and U.S. Patent Application No. 17 / 095,239, filed November 11, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The subject of this disclosure relates to overlay processing for immersive teleconferencing and telepresence for remote terminals (ITT4RT), and more specifically, to signaling for overlaying omnidirectional video and images, such as viewing a presentation / screen-sharing stream or 2D video overlaid on top of a 360-degree video stream. Background Technology

[0004] When using omnidirectional media streaming, only the portion of the content corresponding to the user's viewport is rendered when the user is using a head-mounted display (HMD) to provide the user with a realistic view of the media stream.

[0005] Figure 1 illustrates a traditional scenario (Scenario 1) of an immersive remote conferencing call, where a call is established between A (101), B (102), and C (103). Here, A represents a conference room with an omnidirectional camera (104), and B and C are remote participants using an HMD and a mobile device, respectively. In this scenario, participants B and C send their viewport orientations to A, which in turn sends viewport-related streams to B and C.

[0006] Figure 2a illustrates an extended scenario (Scenario 2) consisting of multiple conference rooms (2a01, 2a02, 2a03, 2a04). User B (2a06) watches video using an HMD, and user C (2a07) watches the stream using a mobile device. B and C send their viewport orientations to the conference rooms, which in turn send viewport-dependent streams to B and C. Another scenario is one where a call is established using a Media Resource Function (MRF) / Media Control Unit (MCU) (b05), as shown in Figure 2b. The MRF and MCU are multimedia servers that provide media-dependent functionality for bridging terminals in a multi-party conference call. Here, multiple conference rooms send their respective videos to the MRF / MCU. These videos are viewport-independent; that is, the entire 360-degree video is sent to the media server regardless of the user's viewport for a particular video being streamed. The media server receives the viewport orientations of users (B (2b06) and C (2b07)) and sends viewport-dependent streams to them accordingly.

[0007] Additionally, in this extended scenario, remote users can select one of several available 360-degree videos to watch from conference rooms (2a01 to 2a04, or 2b01 to 2b04). In this case, users send information about the video they want to stream and its viewport orientation to the conference room or MRF / MCU. Users can trigger a switch from one room to another based on active speakers. Furthermore, the media server can pause receiving video streams from any conference room without any active users.

[0008] ISO 23090-2 defines overlay as "a segment of visual media rendered on or in a viewport over an omnidirectional video or omnidirectional image project." Referring now back to Figure 2a / Figure 2b, when any participant in conference room A is sharing any presentation, that presentation is broadcast as a stream to other users in addition to being displayed in conference room A. This stream can be overlaid on top of a 360-degree video. Furthermore, overlay can also be used for 2D streams.

[0009] Two types of overlay rendering can be defined for use in ITT4RT:

[0010] Viewport-dependent overlay

[0011] Sphere-related two-dimensional superposition

[0012] The following parameters conforming to the OMAF specification can be defined for "sphere-related two-dimensional superposition":

[0013] overlay_ID, overlay_azimuth, overlay_elevation, overlay_tilt, overlay_azimuth_range, overlay_elevation_range, overlay_rot_yaw, overlay_rot_pitch, overlay_rot_roll, region_depth_minus1, timeline_change_flag and name

[0014] For "viewport-dependent overlay", the following parameters can be defined for ITT4RT:

[0015] overlay_ID, overlay_rect_left_percent, overlay_rect_top_percent, overlay_rect_width_percent, overlay_rect_height_percent, relative_disparity_flag, disparity_in_percent, disparity_in_pixels and name

[0016] Regarding user interaction with the overlay, the overlay can obviously include the following additional parameters:

[0017] change_position_flag, change_depth_flag, switch_on_off_flag, change_opacity_flag, resize_flag, rotation_flag, change_position_flag, change_depth_flag, switch_on_off_flag, change_opacity_flag, resize_flag, and rotation_flag Summary of the Invention

[0018] This disclosure relates to methods, systems, and computer-readable media for video conferencing. According to one aspect, a method for video conferencing is provided. The method may include: receiving video data associated with a session of an immersive remote conference; identifying parameters associated with the video data, wherein the parameters specify overlay data associated with the session of the immersive remote conference; and displaying video data having one or more overlays based on the identified parameters.

[0019] According to another aspect, a computer system for video conferencing is provided. The computer system may include one or more processors, one or more computer-readable storage devices, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, enabling the computer system to perform a method. The method may include: receiving video data associated with a session of an immersive remote conference; identifying parameters associated with the video data, wherein the parameters specify overlay data associated with the session of the immersive remote conference; and displaying video data having one or more overlays based on the identified parameters.

[0020] According to yet another aspect, a computer-readable medium for video conferencing is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions, executable by the processor, can perform a method that accordingly may include: receiving video data associated with a session of an immersive remote conference; identifying parameters associated with the video data, wherein the parameters specify overlay data associated with the session of the immersive remote conference; and displaying video data having one or more overlays based on the identified parameters. Attached Figure Description

[0021] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which is read in conjunction with the accompanying drawings. The various features in the drawings are not drawn to scale and are illustrated for clarity by those skilled in the art in conjunction with the detailed description. In the drawings:

[0022] Figure 1 is a schematic diagram of the ecosystem of immersive remote conferencing.

[0023] Figure 2a is a schematic diagram of a multi-party, multi-meeting room remote conference without MRF / MCU.

[0024] Figure 2b is a schematic diagram of a multi-party, multi-meeting room remote conference with MRF / MCU.

[0025] Figures 3a and 3b illustrate multi-party, multi-conference remote conferencing with overlays from a single sender, without an MRF / MCU.

[0026] Figure 4a illustrates a multi-party, multi-conference remote conference without MRF / MCU, which includes overlays from multiple senders.

[0027] Figure 4b illustrates a multi-party, multi-conference remote conference with MRF / MCU, including overlays from multiple senders.

[0028] Figure 5 This is an operation flowchart illustrating the steps performed by the program used for immersive remote conferencing.

[0029] Figure 6 This is a schematic diagram of a computer system. Detailed Implementation

[0030] This document discloses detailed embodiments of the claimed structures and methods. However, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be embodied in various forms. These structures and methods may be embodied in many different forms and should not be construed as limiting themselves to the exemplary embodiments described herein. However, these exemplary embodiments are provided so that this disclosure is thorough and complete and fully conveys the scope of protection to those skilled in the art. In this description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0031] Throughout this description, reference is made to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0032] This disclosure introduces an overlay parameter to limit the maximum number of overlays that a user at the receiving end can use.

[0033] This disclosure introduces an overlay parameter to limit the list of overlays that the sender is sharing.

[0034] This disclosure introduces an overlay parameter to limit the overlay list that the sender is sharing and that the user at the receiver wants to use.

[0035] This disclosure introduces an overlay parameter to limit whether a user at the receiving end is allowed to overlay a stream from a source other than the stream shared by the sender of the 360-degree video on top of the 360-degree video.

[0036] This disclosure introduces an overlay parameter to limit the list of allowed senders, which the user at the receiver can use together with the overlay shared by the senders of the 360-degree video.

[0037] For immersive remote conferencing, when overlaying video or images onto 360-degree video, information such as the following should be included:

[0038] Overlay source, specifying the image or video to be used for overlay.

[0039] Overlay rendering type, describing whether the overlay is fixed relative to the viewport or relative to the sphere.

[0040] Rendering properties such as opacity

[0041] User interaction attributes

[0042] Referring back to Figures 2a and 2b, multiple meeting rooms equipped with omnidirectional cameras are in the middle of a remote meeting, and the user selects a video stream from one of the meeting rooms to be displayed as an immersive video. Now, when any other presentation material / shared screen is used with the 360-degree video that the user is streaming, that presentation material / shared screen is sent as a separate stream and overlaid on top of the 360-degree video.

[0043] In one embodiment, referring to Figures 3a and 3b, users (3a01, 3b01) are streaming immersive video from remote conference room A (3a02, 3b02) onto their HMDs. Conference room A uses screen sharing to display video streams from conference rooms X (3a03, 3b03), Y (3a04, 3b04), and Z (3a05, 3b05), where X is streaming a 2D video stream, while Y and Z are streaming presentation streams. The streams from conference rooms X, Y, and Z are also broadcast to all other remote users. Users can use the parameter "max_overlay," which can be defined as the maximum number of overlays a user can support. The value of this parameter can be based on the user's resource availability. This capacity can be provided in the Session Description Protocol (SDP) during the initial offer-response negotiation, or it can be negotiated during the session based on changes in the user's resource availability (e.g., device battery consumption, bandwidth availability, etc.). The case of max_overlay=0 is reserved for scenarios where the receiver does not support any overlays. Figure 3b depicts the same situation when the MRF / MCU (3b06) is used to establish a call.

[0044] In the same or another embodiment, an additional parameter "list_overlay" is defined, which may include a list of overlays being shared by the sender (by listing the overlay identifiers, overlay_ids, of these overlays). This list of overlays may be sent by the sender to the receiver. The receiver can select overlays to stream from the list and send a reduced list back to the sender. This parameter may be negotiated during the initial offer-response negotiation or renegotiated during the session and provided in the SDP. The case of list_overlay=0 is reserved for scenarios where the receiver does not support overlays (i.e., max_overlay equals 0). The total number of overlays sent by the sender to the user should be less than the value of max_overlay.

[0045] In the same or another embodiment, in addition to sending the overlay list, the sender may also send an overlay priority, which may be included in the parameter "list_overlay". This priority is set by the sender based on the content of the stream. For example, any supporting material used for the presentation will be given a higher priority than 2D video from any other conference room. In addition, the sender may optionally send the bandwidth and decoding computation requirements for each overlay in the multiple overlays. `list_overlay` equal to 0 is reserved for scenarios where the sender does not send any overlay list to the user (therefore, the user will not receive any overlays). Once the overlay list is received from the sender, the user will reply with a list of overlays they can support. This reply may be based on the overlay priority and overlay characteristics sent by the sender, as well as the receiver's computational resources and available network bandwidth. This can be specified under the parameter "list_overlay". These parameters may be negotiated during the initial offer-response negotiation or during the session and are provided in the SDP. Consider a scenario where the user's bandwidth is reduced during the session. When this scenario occurs, the parameter "max_overlay" and optionally the parameter "list_overlay" can be renegotiated. As a result, the value of list_overlay may decrease.

[0046] In another embodiment, referring to Figure 4a, consider the following scenario: A client wants to use a 360-degree video stream from A, but wants to use overlays from Y (4a304) and Z (4a305) (the video streams of Y and Z are not shared by A). In this scenario, the client needs to know whether the sender of the 360-degree video stream allows the user to use overlays from other sources to stream on its video (i.e., along with the 360-degree video stream from A). For this purpose, the parameter "use_other_overlay_flag" can be added. When use_other_overlay_flag is set to 1, it specifies that the user is allowed to use overlays from other senders that are not shared by the sender of the 360-degree video. The value of this parameter can be set by the sender that is streaming the 360-degree video. The same situation is depicted in Figure 4b when the MRF / MCU (4b06) is used to establish a call. A value of 0 for "use_other_overlay_flag" means that the receiver is not allowed to use overlays from any other sender with the video stream of that sender.

[0047] In the same or another embodiment, when `use_other_overlay_flag` equals 1, the sender can send the receiver a list of other senders whose overlays can be used with the 360-degree video stream. This can be limited using the parameter `list_allowed_sender_overlays`. When `use_other_overlay_flag` is 0, users are not allowed to overlay streams from other remote users / conference rooms that are not shared by the sender of the 360-degree video stream.

[0048] In the same or another embodiment, the above parameters can be used to render sphere-related 2D overlays and viewport-related overlays.

[0049] The overlay processing techniques described above for immersive remote conferencing and telepresence can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 6 A computer system 600 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0050] Computer software can be coded using any suitable machine code or computer language, which can be assembled, compiled, linked or similar mechanisms to create code containing instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU) or via interpretation, microcode execution or other means.

[0051] These instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0052] Now for reference Figure 5 The flowchart illustrates a program-executed method 500 for matching queries based on co-attention score.

[0053] At 602, method 500 includes receiving video data associated with a session of an immersive remote conference.

[0054] At 604, method 500 includes identifying parameters associated with video data that specify overlay data associated with a session of an immersive remote conference.

[0055] At 606, method 500 includes displaying video data with one or more overlays based on the identified parameters.

[0056] Understandable Figure 5 This is merely a description of one implementation method and does not imply any limitation on how different implementation methods can be carried out. Many modifications can be made to the depicted environment to suit design and implementation requirements.

[0057] In some embodiments, this parameter is provided in the Session Description Protocol (SDP).

[0058] In some embodiments, the parameter is negotiated during the initial offer-response negotiation or during a session of an immersive remote conference.

[0059] In some embodiments, this parameter limits the maximum number of stacks and constrains the number of stacks at a specific time.

[0060] In some embodiments, this parameter defines a list of overlay identifiers shared by the sender at a given point in time.

[0061] In some embodiments, the parameter is limited to a list of overlay identifiers shared by the sender and supported by the receiver at a given point in time.

[0062] In some embodiments, this parameter defines whether the receiver is allowed to use an overlay other than the overlay shared by the sender.

[0063] In some embodiments, this parameter defines a list of allowed senders from which the receiver receives and uses the overlay.

[0064] In some embodiments, the received video data is associated with a sender, and the parameter defines whether the receiver is allowed to use overlays from another sender that is not associated with the received video data.

[0065] In some embodiments, this parameter defines a priority list of overlay identifiers shared by the sender that the receiver is allowed to use.

[0066] Figure 6 The components of the computer system 600 shown are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of components shown in the exemplary embodiments of the computer system 600.

[0067] Computer system 600 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to one or more human users through input such as: tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

[0068] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard 501, mouse 502, touchpad 503, touch screen 510, data glove (not shown), joystick 505, microphone 506, scanner 507, camera 508.

[0069] Computer system 600 may also include certain human-machine interface output devices. Such human-machine interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback from touchscreen 510, data gloves (not shown), or joystick 505, but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers 509, headphones (not shown)), and visual output devices (e.g., screen 510 including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input functionality, each screen may or may not have tactile feedback functionality—some of these screens are capable of outputting two-dimensional or more three-dimensional visual outputs through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0070] Computer system 600 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 520 with media such as CD / DVD 521, finger drives 522, removable hard disk drives or solid-state drives 523, conventional magnetic media such as magnetic tapes and floppy disks (not shown), devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown), etc.

[0071] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0072] Computer system 600 may also include interfaces to one or more communication networks. These networks can be, for example, wireless networks, wired networks, or optical networks. Further, networks can be local area networks, wide area networks, metropolitan area networks, vehicle and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require connection to external network interface adapters (e.g., USB ports of computer system 600) via certain general-purpose data ports or peripheral buses 449; other network interfaces are typically integrated into the core of computer system 600 via connection to the system bus (e.g., an Ethernet interface in a PC computer system or a cellular network interface in a smartphone computer system). Computer system 600 can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0073] The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel 540 of the computer system 600.

[0074] The core 540 may include one or more CPUs 541, GPUs 542, dedicated programmable processing units in the form of Field Programmable Gate Areas (FPGAs) 543, hardware accelerators 544 for certain tasks, etc. These devices, along with read-only memory (ROM) 545, random access memory 546, and internal mass storage 547 such as internal non-user-accessible hard disk drives (SDs), etc., can be connected via a system bus 548. In some computer systems, the system bus 548 can be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices can be directly connected to the core's system bus 548 or connected to the core's system bus via a peripheral bus 549. Peripheral bus architectures include PCI, USB, etc.

[0075] CPU 541, GPU 542, FPGA 543, and accelerator 544 can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM 545 or RAM 546. Transient data can also be stored in RAM 546, while permanent data can be stored, for example, in internal mass storage 547. Fast storage and retrieval of any storage device can be achieved using a cache, which can be closely associated with one or more CPUs 541, GPUs 542, mass storage 547, ROM 545, RAM 546, etc.

[0076] Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0077] As a non-limiting example, a computer system having architecture 500, particularly kernel 540, can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and some non-transitory memory of kernel 540, such as mass storage 547 or ROM 545 within the kernel. Software implementing various embodiments of this disclosure can be stored in such means and executed by kernel 540. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause kernel 540, particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 546 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., accelerator 544), which may replace or operate with the software to perform a particular process or a specific portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0078] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail integration. A computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform operations.

[0079] Computer-readable storage media can be tangible means for holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanical encoding means such as raised structures in recesses where instructions are recorded, and any suitable combination of the foregoing. The computer-readable storage media used herein should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through optical fibers), or electrical signals transmitted through wires.

[0080] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0081] Computer-readable program code / instructions used to perform operations can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). Computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, an FPGA, or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuitry for performing various aspects or operations by utilizing the status information of the computer-readable program instructions.

[0082] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0083] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, implement the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or instruction, including one or more executable instructions for implementing one or more specified logical functions. The method, computer system, and computer-readable medium may include other, fewer, different, or differently arranged blocks compared to those depicted in the figures. In some alternative implementations, the functions marked in the blocks may occur in a non-consecutive order. For example, depending on the functions involved, two consecutively shown blocks may actually execute simultaneously or substantially simultaneously, or sometimes in reverse order. It will also be noted that each block shown in the block diagrams and / or flowcharts, and combinations of blocks shown in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or performs a combination of dedicated hardware and computer instructions.

[0085] It is evident that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to these implementations. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0086] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, the indefinite article used herein is intended to include one or more items and may be used interchangeably with “one or more.” Additionally, the term “set” used herein is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more.” If the intention is to use an item, the term “a” or similar language is used. Furthermore, the terms “having,” “possessing,” “with,” etc., used herein are intended as open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

[0087] The description of various aspects and embodiments is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Even though combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically described in the claims and / or not specifically disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with all other claims in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for video conferencing, executed by a processor, the method comprising: Receive video data associated with sessions in immersive remote meetings; Identify parameters associated with the video data, wherein the parameters specify overlay data associated with a session of the immersive remote conference; and Based on the identified parameters, the video data with one or more overlays is displayed, wherein the received video data is associated with a sender, and the parameters define whether the receiver is allowed to use an overlay from another sender not associated with the received video data.

2. The method according to claim 1, wherein, The parameters are provided in the Session Description Protocol (SDP).

3. The method according to claim 1, wherein, The parameters are negotiated during the initial offer-response negotiation or during the session of the immersive remote conference.

4. The method according to any one of claims 1-3, wherein, The parameters limit the maximum number of stacks and constrain the number of stacks at a specific time.

5. The method according to any one of claims 1-3, wherein, The parameter defines the list of overlay identifiers shared by the sender at a given point in time.

6. The method according to any one of claims 1-3, wherein, The parameters are limited to a list of overlay identifiers shared by the sender and supported by the receiver at a given point in time.

7. The method according to any one of claims 1-3, wherein, The parameters define a list of allowed senders, and the receiver receives and uses the overlay from senders in the list.

8. The method according to any one of claims 1-3, wherein, The parameter defines the priority list of overlay identifiers shared by the sender that the receiver is allowed to use.

9. A computer system for video conferencing, the computer system comprising: One or more computer-readable non-transitory storage media are configured to store computer program code; as well as One or more computer processors are configured to access and be instructed by the computer program code to perform operations, the computer program code including: A receiving code is configured to cause the one or more computer processors to receive video data associated with a session of an immersive remote conference; An identification code is configured to cause the one or more computer processors to identify parameters associated with the video data, wherein the parameters specify overlay data associated with a session of the immersive remote conference; and Display code is configured to cause the one or more computer processors to display video data with one or more overlays based on identified parameters, wherein the received video data is associated with a sender, and the parameters define whether the receiver is allowed to use an overlay from another sender not associated with the received video data.

10. The computer system according to claim 9, wherein, The parameters are provided in the Session Description Protocol (SDP).

11. The computer system according to claim 9, wherein, The parameters are negotiated during the initial offer-response negotiation or during the session of the immersive remote conference.

12. The computer system according to any one of claims 9-11, wherein, The parameters limit the maximum number of stacks and constrain the number of stacks at a specific time.

13. The computer system according to any one of claims 9-11, wherein, The parameter defines the list of overlay identifiers shared by the sender at a given point in time.

14. The computer system according to any one of claims 9-11, wherein, The parameters are limited to a list of overlay identifiers shared by the sender and supported by the receiver at a given point in time.

15. The computer system according to any one of claims 9-11, wherein, The parameter defines a list of allowed senders, wherein the receiver receives and uses the overlay from senders in the list of senders.

16. A non-transitory computer-readable medium having a computer program for video conferencing stored thereon, the computer program being configured to cause one or more computer processors to perform the method according to any one of claims 1-8.

17. An apparatus for video conferencing, the apparatus comprising: The receiving module is configured to receive video data associated with a session of an immersive remote conference; A recognition module is configured to recognize parameters associated with the video data, wherein the parameters specify overlay data associated with a session of the immersive remote conference; and The display module is configured to display video data with one or more overlays based on identified parameters, wherein the received video data is associated with a sender, and the parameters define whether the receiver is allowed to use an overlay from another sender not associated with the received video data.

Images