Methods, devices, and media for streaming video data

Abstract

Methods, non-transitory computer-readable media, and computer systems for processing a video stream are provided. The method can include receiving a 360° video stream, the 360° video stream including a multi-overlay parameter; determining that a user terminal is allowed to overlay the 360° video stream with two or more overlay objects when the multi-overlay parameter is a first value, and displaying, by the user terminal, the 360° video stream with the two or more overlay objects; and determining that the user terminal is not allowed to overlay the 360° video stream with two or more overlay objects when the multi-overlay parameter is a second value, and displaying the 360° video stream with zero overlay objects or one overlay object.

Description

[0001] Priority information

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 064,252, filed August 11, 2020, and U.S. Patent Application No. 17 / 098,388, filed November 15, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of data processing, and more specifically to a method, apparatus, user terminal, and non-transitory computer-readable medium for streaming video data. Background Technology

[0004] An overlay object is a piece of visual media rendered on an omnidirectional video, image item, or viewport.

[0005] When using omnidirectional media streaming, only the portion of the content corresponding to the user's viewport is rendered, while a head-mounted display (HMD) device, such as one that provides the user with a realistic view of the media stream, is used.

[0006] However, existing technologies lack interactive multi-overlay processing methods for immersive teleconferencing and telepresence for remote terminals (ITT4RT). Summary of the Invention

[0007] Implementations relate to methods for streaming video data, user terminals, computer devices, and non-transitory computer-readable media. According to one aspect, a method for streaming video data can be provided. The method may include: receiving a 360° video stream from an electronic device or server by a user terminal, wherein the 360° video stream includes multiple overlay parameters indicating whether the user terminal allows the 360° video stream to be overlaid with two or more overlay objects; displaying the 360° video stream by the user terminal; when the multiple overlay parameters are a first value, determining by the user terminal that the user terminal allows the 360° video stream to be overlaid with two or more overlay objects, and displaying the 360° video stream having two or more overlay objects; and when the multiple overlay parameters are a second value, determining by the user terminal that the user terminal does not allow the 360° video stream to be overlaid with two or more overlay objects, and displaying the 360° video stream having zero overlay objects or one overlay object.

[0008] The first value may be different from the second value, and one or more overlay objects may include one or more of the following: another video stream or image.

[0009] A 360° video stream may include an overlap parameter that indicates whether one of two or more overlap objects is allowed to overlap with a first overlap object.

[0010] The method may further include: when an overlap parameter indicates that one of two or more overlap objects is allowed to overlap with a first overlap object, a user terminal displays the 360° video stream having the two or more overlap objects, wherein the first overlap object overlaps with one of the two or more overlap objects.

[0011] A 360° video stream can include content-based attributes that allow the use of two or more overlay objects.

[0012] A 360° video stream may include a flag that signals when a user is allowed to share the streamer’s information with other senders.

[0013] The method may further include: receiving a request from an electronic device or server by a user terminal, wherein the request is a request for sharing media stream attributes among one or more users in a remote conference; and sending information from the user terminal to the electronic device or server regarding the media stream attributes authorized for sharing by the user terminal.

[0014] Information about the attributes of the media stream that a user is willing to share includes an identifier (ID) corresponding to the sender ID.

[0015] The method may also include: assigning a unique overlay ID by an electronic device or server by combining at least one sender ID with at least one overlay ID.

[0016] The method may also include negotiating multiple stacked parameters during the initial proposal-response negotiation or during the session.

[0017] Multiple overlay parameters can be proposed in the Session Description Protocol (SDP) for 360° video streams.

[0018] The method may also include negotiating overlapping parameters during the initial proposal-response negotiation or during the session.

[0019] Overlapping parameters can be proposed in the Session Description Protocol (SDP) of 360° video streams.

[0020] The method may also include negotiating content-based attributes during the initial proposal-response negotiation or during the session.

[0021] Content-based attributes are proposed in the Session Description Protocol (SDP) for 360° video streams.

[0022] The method may also include: transmitting from the user terminal to an electronic device or server whether another user is authorized to share streaming information shared by the user terminal with other electronic devices.

[0023] The method may also include: transmitting from the user terminal to an electronic device or server whether another user is authorized to share streaming information shared by the user terminal with other electronic devices.

[0024] The method may further include: sharing at least one additional overlay object with other electronic devices, other than any one or more overlay objects generated by the electronic device or server, wherein the at least one additional overlay object includes only overlay objects in which the sender of the corresponding overlay video has authorized the sharing of the overlay object.

[0025] According to an embodiment, a computer device may be provided, which may include: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access the computer program code and execute the above-described method for streaming video data as instructed by the computer program code.

[0026] According to an embodiment, a user terminal for streaming video data can be provided, which may include: a communication interface; a display; a receiving unit; a display unit; a multi-overlay licensed unit; and a multi-overlay unlicensed unit.

[0027] The receiving unit is configured to receive a 360° video stream via a communication interface, wherein the 360° video stream includes a multi-overlay parameter indicating whether the terminal is permitted to overlay the 360° video stream with two or more overlay objects. The display unit is configured to control a display to show the received 360° video stream. A multi-overlay authorization unit is configured to perform the following operations: when the multi-overlay parameter is a first value, determine that the user terminal is permitted to overlay the 360° video stream with two or more overlay objects, and control the display to show the 360° video stream with two or more overlay objects. A multi-overlay unauthorization unit is configured to perform the following operations: when the multi-overlay parameter is a second value, determine that the user terminal is not permitted to overlay the 360° video stream with two or more overlay objects, and control the display to show the 360° video stream with zero or one overlay object.

[0028] The first value may be different from the second value, and one or more overlay objects may include one or more of the following: another video stream or image.

[0029] The 360° video stream may include an overlap parameter indicating whether it is permissible to overlap one of two or more overlap objects with a first overlap object. The user terminal may also include an overlap authorization unit configured to perform the following operations: when the overlap parameter indicates that overlap between one of two or more overlap objects and the first overlap object is permitted, control the display to display the 360° video stream having the two or more overlap objects, wherein the first overlap object overlaps with one of the two or more overlap objects.

[0030] According to an embodiment, a non-transitory computer-readable medium storing a computer program for streaming video data may be provided, the computer program being configured to cause one or more computer processors to execute the above-described method for streaming video data.

[0031] According to the method, apparatus, user terminal and non-transitory computer-readable medium for streaming video data provided by the present invention, an interactive multi-overlay processing method for immersive remote conferencing and remote presentation for remote terminals is provided, such that signaling for multiple interactive overlay objects for omnidirectional video and images is displayed as overlay objects on a 360° video stream being displayed, for example in a presentation / screen sharing stream or two-dimensional (2D) video. Attached Figure Description

[0032] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. The various features in the drawings are not drawn to scale because these illustrations are intended to clearly facilitate understanding by those skilled in the art in conjunction with the detailed description. In the drawings:

[0033] Figure 1 This is an illustrative representation of the ecosystem used for immersive remote conferencing;

[0034] Figure 2A This is an illustrative illustration of multi-party, multi-meeting room remote conferencing without using MRF / MCU;

[0035] Figure 2B This is an illustrative illustration of using MRF / MCU for remote conferencing in multiple meeting rooms.

[0036] Figure 3A The use of the parameter "use_multiple_flag" is shown for multi-party multi-room remote conferencing using multiple overlay objects from a single sender without using MRF / MCU;

[0037] Figure 3B The use of the parameter "use_multiple_flag" is shown for remote multi-party multi-meeting rooms using multiple overlay objects from a single sender when using MRF / MCU;

[0038] Figure 4A This demonstrates the content-based use of additional overlay objects for multi-party, multi-conference room remote conferencing using multiple overlay objects from multiple senders without using MRF / MCU;

[0039] Figure 4B The following is illustrated, according to an embodiment, the content-based use of additional overlay objects for multi-party, multi-conference room remote conferencing using multiple overlay objects from multiple senders in the context of using MRF / MCU;

[0040] Figure 5 This is an illustrative description of a computer system according to an implementation method; and

[0041] Figure 6 An operational flowchart illustrating the steps of a method for processing video conferencing is shown. Detailed Implementation

[0042] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that can be implemented in various forms. These structures and methods can be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the specification to avoid unnecessarily obscuring the presented embodiments.

[0043] The implementation methods generally relate to the field of data processing, and more specifically to video conferencing / remote conferencing using omnidirectional video. Furthermore, the exemplary implementations described below provide systems, methods, and computer programs for processing video conferencing / remote conferencing using omnidirectional video.

[0044] This document will describe aspects with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It should be understood that each block in the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0045] Figure 1A scenario of an immersive remote conference call according to an embodiment is illustrated, wherein a call is organized between room A (101), user B (102), and user C (103). According to the embodiment, room A represents a conference room with an omnidirectional (360°) camera device (104), and user B and user C are remote participants using an HMD device and a mobile device, respectively. In this scenario, remote conference call participants user B and user C can send information to room A about their respective viewport orientations, and room A can then send back viewport-dependent streams to them.

[0046] Figure 2A Another scenario of an immersive remote conference call, according to an embodiment, is illustrated, involving multiple conference room participants (room A (2a01), room X (2a02), room Y (2a03), and room Z (2a04)). According to the embodiment, user B (2a06) can use an HMD display to view a video stream (e.g., a 360° video stream), and user C (2a07) can use a mobile device to view the video stream. Users B and C can send their viewport orientations to the main room (e.g., room A), which in turn sends viewport-dependent streams to them.

[0047] According to the implementation method, Figure 2B Another scenario is illustrated where a video call is established using a Media Resource Function (MRF) / Media Control Unit (MCU) (2b05), where the MRF / MCU is shown as a multimedia server providing media-related functions for bridging remote terminals in a multi-party conference call. However, the MRF / MCU can be another electronic device besides a server. According to an implementation, electronic devices corresponding to each conference room can send their respective videos to the MRF / MCU. These videos are viewport-independent (i.e., the entire 360° video is sent to the MRF / MCU (e.g., the media server), regardless of the viewport of the user's streaming video). The media server can receive the viewport orientations of user B (2b06) and user C (2b07) and send corresponding viewport-dependent streams to user B and user C accordingly.

[0048] Furthermore, in an extended scenario, remote users can select one of the available 360° videos to watch from conference rooms (2a01 to 2a04 / 2b01 to 2b04). In this case, the user sends information (e.g., a request) to the conference room or MRF / MCU regarding the video they want to stream and its viewport orientation. According to an implementation, the user can trigger a switch from one room to another based on an active speaker. That is, according to an implementation, the user terminal providing the current audio output can be used to determine the viewport orientation such that the viewport orientation corresponds to the active speaker. Additionally, the media server can pause receiving video streams from any conference room that does not have any active users.

[0049] An overlay object is a piece of visual media rendered on an omnidirectional video, image item, or viewport.

[0050] Return to reference Figure 2A and Figure 2B When a presentation is shared by any participant in conference room A, the presentation can be broadcast as a stream to other users (e.g., remote terminals associated with the user, such as smartphones or HMD devices) in addition to being displayed in conference room A.

[0051] According to one implementation, the stream can be overlaid on a 360° video. According to another implementation, the overlay object can also be used for a 2D stream.

[0052] According to the implementation, at least two types of overlay rendering can be defined for ITT4RT. The first type can be viewport-dependent overlay, and the second type can be sphere-dependent 2D overlay.

[0053] According to the implementation, for "viewport-dependent overlay", one or more of the following parameters can be defined for ITT4RT: overlay_ID, overlay_rect_left_percent, overlay_rect_top_percent, overlay_rect_width_percent, overlay_rect_height_percent, relative_disparity_flag, disparity_in_percent, disparity_in_pixels, and name.

[0054] According to the implementation, one or more parameters can be defined for “sphere-related two-dimensional overlay” consistent with the Open Mobile Access Abstract Framework (OMAF) specification, such as: overlay_ID, overlay_azimuth, overlay_elevation, overlay_tilt, overlay_azimuth_range, overlay_elevation_range, overlay_rot_yaw, overlay_rot_pitch and overlay_rot_roll, region_depth_minus1, timeline_change_flag, and name.

[0055] Regarding user interaction with overlay objects, overlay objects may specifically include one or more of the following parameters: change_position_flag, change_depth_flag, switch_on_off_flag, change_opacity_flag, resize_flag, rotation_flag, change_position_flag, change_depth_flag, switch_on_off_flag, change_opacity_flag, resize_flag or rotation_flag, use_other_overlay_flag.

[0056] According to the implementation, an overlay parameter can be used to specify whether a user is allowed to use more than one overlay object on top of a 360° video.

[0057] According to the implementation, the overlay parameter can be used to define whether users are allowed to overlap two or more overlay objects.

[0058] According to the implementation method, the overlay parameter can be used to define a content-based attribute, and the "user_other_overlay_flag" can be used based on this content-based attribute.

[0059] According to the implementation, by using a combination of a sender identifier (ID) and (one or more) overlay IDs, or by constructing a unique ID based on the sender ID and (one or more) overlay IDs, the sender can obtain information about other overlay objects in a remote conference to know the possible combinations, and signal to other senders which overlay objects can be combined with its own video.

[0060] Regarding immersive remote conferencing, when multiple overlay videos or images are superimposed on a 360° video, the following information may be included, such as:

[0061] a. Specify the image or video source to be used as the overlay object;

[0062] b. Describes the overlay rendering type as to whether the overlay object is anchored relative to the viewport or a sphere;

[0063] c. Rendering properties, such as opacity;

[0064] d. User interaction attributes.

[0065] Return to reference Figure 2A and Figure 2B In this scenario, multiple meeting rooms equipped with omnidirectional cameras are in a remote conference, and the user selects a video stream from one of the meeting rooms to display as an immersive video. Now, when any additional presentation materials / shared screens are used in conjunction with the 360° video being streamed by the user, those materials / shared screens are sent as separate streams as overlaid objects on the 360° video.

[0066] According to the implementation method, such as Figure 3A As shown, a user (3a01) is streaming immersive video from remote conferencing room A (3a02) on his electronic device (e.g., HMD). Room A can use screen sharing to display video streams from rooms X (3a03) and Y (3a04), where room X is streaming a 2D video stream and room Y is streaming a presentation stream (e.g., a slideshow or chart). According to the implementation, the streams from rooms X and Y can also be broadcast to all other remote users.

[0067] According to one implementation, multiple overlay flags can be defined for the user to allow the display of more than one overlay object. According to another implementation, the multiple overlay objects may include those from the sender of the 360° video and additional overlay objects from other users.

[0068] When the multiple overlay flag is set to 1, users streaming 360° video are allowed to use multiple overlay objects on top of the 360° video.

[0069] According to the implementation, when the multiple overlay flag is set to 0, the user at the receiver uses only one overlay object. Of course, other values ​​besides 0 and 1 can be used, but it should be noted that one bit can be used for the multiple overlay flag to use 0 and 1.

[0070] According to the implementation method, the multiple overlay flag value can be set by the sender of the 360° video.

[0071] According to the implementation, multiple overlay flags can be proposed in the Session Description Protocol (SDP) during the initial proposal-response negotiation, and multiple overlay flags can also be renegotiated during the session.

[0072] According to another implementation, such as Figure 3B As shown, when the MRF / MCU (3b06) is used to establish a call, the same or similar usage of multiple overlay flags can be used.

[0073] In the same or another implementation, additional flags, such as overlapping flags, may be defined.

[0074] According to the implementation, the overlay flag can indicate whether a user is allowed to overlay an overlay object from another sender onto an overlay object shared by the sender of the 360° video.

[0075] According to the implementation method, the overlay flag can be equal to 0 by default.

[0076] According to the implementation, when the overlay is set to 1, the user can overlay an overlay object from another sender onto an overlay object from the sender of the 360° video.

[0077] According to the implementation, the parameters can be set by the sender of the 360° video, and the parameters can be proposed during the initial proposal-response negotiation, and the parameters can also be renegotiated during the session.

[0078] According to the same or another implementation method, refer to Figure 4A Consider the following scenario: In this scenario, a client (e.g., user 4a01) wants to use a 360° video stream from room A (room A uses screen sharing to display the stream from room X), and also wants to use an overlay object from room Z (4a05), whose stream is not shared by room A (except for simultaneously compressing the stream shared by room A).

[0079] According to the implementation, the flag "use_other_overlay_flag" can be used to specify whether users are allowed to use overlay objects from other senders that are not shared by the sender of the 360° video.

[0080] According to the implementation, the value of the flag "use_other_overlay_flag" can be set by the sender who is streaming its 360° video.

[0081] According to the implementation, "use_other_overlay_flag" can be based on the content type of the overlay object that the sender of the 360° video is sharing.

[0082] According to the implementation, the content type can be defined in the SDP signaling during the initial proposal response negotiation by including the attribute "a = permissible_content" under line m= for each overlay object shared by the sender.

[0083] According to the implementation method, similar or identical content types that define and use overlay tags can be used for Figure 4B One or more implementations for establishing video calls.

[0084] In the same or another implementation, when initiating a remote conference call, the sender of the 360° video can send a request to all other users in the conference, inquiring about their reception of media stream attributes. This is to understand the content types of all overlay objects that each user can receive in the remote conference. Users may wish to share or not share their information with the sender. This can be based on the user's security considerations.

[0085] According to the implementation method, the sender can signal to the user whether they are also permitted to share the information with other senders / users. When not permitted, the user can choose not to share the sender's stream with other senders.

[0086] According to the implementation, when a sender receives a reply from a user that includes information from other senders, the sender can use each sender ID along with each overlay ID to assign (one or more) globally unique overlay IDs to all overlay objects in the remote conferencing session.

[0087] According to the implementation method, the construction of a globally unique ID can be shared with all senders and users, so that the globally unique ID can be parsed and processed by all entities.

[0088] According to the same or similar implementation, when a sender allows a user to share their video streams with other senders, the sender may set the “share_streams_info_allowed” flag when communicating with the user (e.g., transmitting to the user).

[0089] In the same implementation, if the user accepts the sender's request to share its attributes, the flags "multiple_overlay_flag" and "overlay_overlap_flag" may also be included to indicate whether other overlay objects may be used with its media stream.

[0090] In the same or similar implementation, when one or more users accept a request for shared user attributes (or sender attributes) from a sender, the following attributes may be included, specifying the permissible content types that may be used with the media stream.

[0091] According to the implementation, after receiving a response from the user about their media stream, the sender may send only selected overlay objects to the user, which may be based on the attributes of other overlay objects.

[0092] While user interaction has been explained in the above description, it should be noted that user interaction is facilitated by electronic devices such as smartphones or HMD devices. Therefore, when the specification describes allowing a user to perform an action, this disclosure applies to a remote terminal associated with the user and being used by the user (e.g., a remote terminal making an inbound video call), which is permitted (or not permitted) to perform the action.

[0093] The overlay processing techniques described above for immersive remote conferencing and telepresence can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 5 A computer system 500 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0094] Computer software can be encoded using any suitable machine code or computer language. Machine code or computer language can be subjected to mechanisms such as assembly, compilation, and linking to create code that includes instructions. These instructions can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0095] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0096] Figure 5 The components shown for computer system 500 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or combination of the components shown in the exemplary embodiments of computer system 500.

[0097] Computer system 500 may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users via, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, tapping), visual input (such as gestures), and olfactory input. Human-machine interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as: audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from still image capturing devices), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0098] According to an embodiment, the input human-machine interface device may include one or more of the following: keyboard 501, mouse 502, touchpad 503, touch screen 510, data glove, joystick 505, microphone 506, scanner 507, or camera device 508.

[0099] According to an embodiment, the computer system 500 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such HMI output devices may include: one or more tactile output devices (e.g., tactile feedback via touchscreen 510, data gloves, or joystick 505, but tactile feedback devices not used as input devices may also exist), audio output devices (such as speakers 509, headphones), visual output devices (such as screens 510, including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, light-emitting diode (LED) (such as organic LED (OLED)) screens, each screen having or not having touchscreen input capability, each screen having or not having tactile feedback capability—some of these screens are capable of outputting two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic graphics output; virtual reality glasses; holographic displays; and cigarette canisters), and printers.

[0100] According to an embodiment, the computer system 500 may also include a human-accessible storage device and its associated medium, such as an optical medium, including an optical disc (CD) read-only memory (ROM) / read / write memory (RW) or a digital versatile disc (DVD) ROM / RW or a combination of CD / DVD ROM / RW 520 and CD / DVD, etc., and medium 521 may be used. Alternatively or additionally, one or more of the following may be used for storage: a thumb drive (522), a removable hard disk drive or solid-state drive (SSD) 523, conventional magnetic media such as magnetic tape and floppy disk, and devices based on application-specific ROM / application-specific integrated circuits (ASICs) / programmable logic devices (PLDs), such as security dongles, etc.

[0101] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the currently disclosed subject matter does not include transmission media, carrier waves, or other transient signals. In other words, computer-readable medium refers to non-transient computer-readable media.

[0102] According to the implementation, the computer system 500 may also include interfaces to one or more communication networks. These networks may be, for example, wireless networks, wired networks, or optical networks. The networks may also be local area, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of networks include: local area networks, such as Ethernet and wireless local area networks (LANs); cellular networks, including Global System for Mobile Communications (GSM), third-generation (3G), fourth-generation (4G), fifth-generation (5G), Long Term Evolution (LTE), or any other communication standard; cable or wireless wide area digital television (TV) networks, including cable TV, satellite TV, and terrestrial broadcast TV; and vehicular and industrial networks, including Controller Area Network Bus (CANBus), etc. Some networks typically require external network interface adapters to be attached to certain general-purpose data ports or peripheral buses (549) (such as, for example, the Universal Serial Bus (USB) port of computer system 500); other networks are typically integrated into the core of computer system 500 by being attached to the system bus as described below (e.g., an Ethernet interface to a personal computer (PC) system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 500 can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcasting TV), one-way transmit-only (e.g., a CAN bus to some CAN bus device), or bidirectional (e.g., using a local area digital network or a wide area digital network to other computer systems). Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0103] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the core 540 of the computer system 500.

[0104] Core 540 may include one or more central processing units (CPUs) 541, graphics processing units (GPUs) 542, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 543, hardware accelerators 544 for certain tasks, etc. These devices, along with read-only memory (ROM) 545, random access memory (RAM) 546, and internal mass storage devices (such as internal non-user-accessible hard disk drives, SSDs, etc.) 547, can be connected via system bus 548. In some computer systems, system bus 548 may be accessed as one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly or via peripheral bus 549 to the core's system bus 548. The peripheral bus architecture may include peripheral component interconnect (PCI), USB, etc.

[0105] CPU 541, GPU 542, FPGA 543, and accelerator 544 can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM 545 or random access memory (RAM) 546. Transient data can also be stored in RAM 546, while permanent data can be stored, for example, in an internal mass storage device 547. Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 541, GPUs 542, mass storage devices 547, ROM 545, RAM 546, etc.

[0106] Computer-readable media may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or the media and computer code may have types known and available to those skilled in the art of computer software.

[0107] By way of example and not limitation, a computer system 500 with an architecture—particularly core 540—can be functionalized by processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of core 540 with non-transitory characteristics, such as internal mass storage device 547 or ROM 545. Software implementing various embodiments of this disclosure can be stored in such devices and executed by core 540. Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause core 540, and particularly its processors (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 546 and modifying such data structures according to the software-defined processes. Alternatively or as an alternative, the computer system may be provided with functionality by hard-wired or otherwise embodied logic in circuitry (e.g., accelerator 544), which may operate in place of or in conjunction with software to perform the specific processing or a specific portion of the specific processing described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry storing software for execution (such as integrated circuits (ICs)), circuitry containing logic for execution, or both. This disclosure covers any suitable combination of hardware and software.

[0108] While some exemplary embodiments have been described in this disclosure, there are changes, substitutions, and various alternatives that fall within the scope of this disclosure. It will therefore be appreciated that, although not explicitly shown or described herein, those skilled in the art will be able to conceive of many systems and methods that embody the principles of this disclosure and are therefore within its spirit and scope.

[0109] Now refer to Figure 6 This document describes an operational flowchart of the steps of a method 600 for video conferencing. In some implementations, one or more process blocks may be executed by a computer and a server. In some implementations, one or more process blocks may be executed by another device or a group of devices, separate from or including the computer and server.

[0110] The implementation methods relate to methods, systems, and non-transitory computer-readable media for video conferencing.

[0111] like Figure 6 As shown, a method for streaming video data can be provided.

[0112] like Figure 6 As shown, the method may include:

[0113] Operation 601: The user terminal receives a 360° video stream from an electronic device or server, wherein the 360° video stream includes multiple overlay parameters that indicate whether the user terminal allows the 360° video stream to be overlaid with two or more overlay objects;

[0114] Operation 602: Display a 360° video stream on the user terminal;

[0115] Operation 603: Determine whether the multi-overlay parameters actually indicate that the user terminal actually allows the 360° video stream to be overlaid with two or more overlay objects.

[0116] Operation 604: When the multiple overlay parameter is the first value, the user terminal determines that it allows the 360° video stream to be overlaid with two or more overlay objects, and the user terminal displays the 360° video stream with two or more overlay objects; and

[0117] Operation 605: When the multi-overlay parameter is the second value, the user terminal determines that the user terminal does not allow the 360° video stream to be overlaid with two or more overlay objects, and the user terminal displays the 360° video stream with zero overlay objects or one overlay object.

[0118] The first value may be different from the second value, and one or more overlay objects may include one or more of the following: another video stream or image.

[0119] A 360° video stream may include an overlap parameter that indicates whether one of two or more overlap objects is allowed to overlap with a first overlap object.

[0120] The method may further include: when an overlap parameter indicates that one of two or more overlap objects is allowed to overlap with a first overlap object, a user terminal displays the 360° video stream having the two or more overlap objects, wherein the first overlap object overlaps with one of the two or more overlap objects.

[0121] A 360° video stream can include content-based attributes that allow the use of two or more overlay objects.

[0122] A 360° video stream may include a flag that signals when a user is allowed to share the streamer’s information with other senders.

[0123] The method may further include: receiving a request from an electronic device or server by a user terminal, wherein the request is a request for sharing media stream attributes among one or more users in a remote conference; and sending information from the user terminal to the electronic device or server regarding the media stream attributes authorized for sharing by the user terminal.

[0124] Information about the attributes of the media stream that a user is willing to share includes an identifier (ID) corresponding to the sender ID.

[0125] The method may also include: assigning a unique overlay ID by an electronic device or server by combining at least one sender ID with at least one overlay ID.

[0126] The method may also include negotiating multiple stacked parameters during the initial proposal-response negotiation or during the session.

[0127] Multiple overlay parameters can be proposed in the Session Description Protocol (SDP) for 360° video streams.

[0128] The method may also include negotiating overlapping parameters during the initial proposal-response negotiation or during the session.

[0129] Overlapping parameters can be proposed in the Session Description Protocol (SDP) of 360° video streams.

[0130] The method may also include negotiating content-based attributes during the initial proposal-response negotiation or during the session.

[0131] Content-based attributes can be proposed in the Session Description Protocol (SDP) for 360° video streams.

[0132] The method may also include: transmitting from the user terminal to an electronic device or server whether another user is authorized to share streaming information shared by the user terminal with other electronic devices.

[0133] The method may also include: transmitting from the user terminal to an electronic device or server whether another user is authorized to share streaming information shared by the user terminal with other electronic devices.

[0134] The method may further include: sharing at least one additional overlay object with other electronic devices, in addition to any one or more overlay objects generated by the electronic device or server, wherein the at least one additional overlay object includes only overlay objects in which the sender of the corresponding overlay video has authorized the sharing of the overlay object.

[0135] According to an embodiment, a computer system for video conferencing can be provided, the computer system including: a communication interface; a display; one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access the computer program code and operate as instructed by the computer program code.

[0136] The computer program code may include: receiving code configured to cause one or more computer processors to receive a 360° video stream via a communication interface, wherein the 360° video stream includes a multi-overlay parameter indicating whether a user terminal allows the 360° video stream to be overlaid with two or more overlay objects; display code configured to cause one or more computer processors to control a display to display the received 360° video stream; multi-overlay authorization code configured to cause one or more computer processors to perform the following operations: when the multi-overlay parameter is a first value, determining that the user terminal allows the 360° video stream to be overlaid with two or more overlay objects, and controlling the display to display the 360° video stream having two or more overlay objects; and multi-overlay unauthorized code configured to cause one or more computer processors to perform the following operations: when the multi-overlay parameter is a second value, determining that the user terminal does not allow the 360° video stream to be overlaid with two or more overlay objects, and controlling the display to display the 360° video stream having zero overlay objects or one overlay object.

[0137] The first value may be different from the second value, and one or more overlay objects may include one or more of the following: another video stream or image.

[0138] The 360° video stream may include an overlap parameter indicating whether it is permissible to overlap one of two or more overlap objects with a first overlap object, and the computer program code may further include overlap authorization code configured to cause one or more processors to perform the following operations: when the overlap parameter indicates that it is permissible to overlap one of two or more overlap objects with a first overlap object, control a display to display the 360° video stream having the two or more overlap objects, wherein the first overlap object overlaps with one of the two or more overlap objects.

[0139] According to an embodiment, a non-transitory computer-readable medium storing a computer program configured to cause one or more computer processors to perform the following operations: receiving a 360° video stream, wherein the 360° video stream includes a multi-overlay parameter indicating whether overlaying the 360° video stream with two or more overlay objects is permitted; controlling a display to display the received 360° video stream; when the multi-overlay parameter is a first value, determining that overlaying the 360° video stream with two or more overlay objects is permitted, and controlling the display to display the 360° video stream having two or more overlay objects; and when the multi-overlay parameter is a second value, determining that overlaying the 360° video stream with two or more overlay objects is not permitted, and controlling the display to display the 360° video stream having zero overlay objects or one overlay object, wherein the first value differs from the second value, and wherein one or more overlay objects include one or more of the following: another video stream or image.

[0140] The 360° video stream may include an overlap parameter indicating whether it is permissible to overlap one of two or more overlap objects with a first overlap object, and the computer program may also be configured to cause one or more computer processors to perform the following operations: when the overlap parameter indicates that it is permissible to overlap one of two or more overlap objects with a first overlap object, control the display to display the 360° video stream having the two or more overlap objects, wherein the first overlap object overlaps one of the two or more overlap objects.

[0141] Understandably, flowcharts (e.g.) Figure 6 This is merely an illustration of one implementation method and does not imply any limitation on how different implementation methods can be implemented. Many modifications can be made to the described environment based on design and implementation requirements.

[0142] The above-described technology can be implemented by one or more processors that can execute computer software with computer-readable instructions, which can be physically stored on one or more computer-readable media (e.g., hard disk drives). For example, Figure 5 A computer system suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0143] Computer software can be encoded using any suitable machine code or computer language. Machine code or computer language can be subjected to mechanisms such as assembly, compilation, and linking to create code that includes instructions. These instructions can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0144] Instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things (IoT) devices, etc.

[0145] While some exemplary embodiments have been described in this disclosure, there are changes, substitutions, and various alternatives that fall within the scope of this disclosure. It will therefore be appreciated that, although not explicitly shown or described herein, those skilled in the art will be able to conceive of many systems and methods that embody the principles of this disclosure and are therefore within its spirit and scope.

[0146] Some implementations may involve systems, methods, and / or computer-readable media at any possible level of technical detail. A computer-readable medium may include one or more computer-readable non-transitory storage media having computer-readable program instructions on it for causing a processor to perform operations.

[0147] A computer-readable storage medium can be a tangible means capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as raised structures in recesses or punch cards recording instructions, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0148] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network such as the Internet, local area network, wide area network, and / or wireless network to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0149] The computer-readable program code / instructions used to perform the operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server as a standalone software package. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the use of the Internet provided by an Internet service provider). In some implementations, electronic circuit systems, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of computer-readable program instructions to personalize the electronic circuits in order to perform aspects or operations.

[0150] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other means to function in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to enable a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, thereby enabling the instructions to be executed on the computer, other programmable apparatus or other device to perform the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or instruction, which includes one or more executable instructions for implementing one or more specified logical functions. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently compared to those depicted in the drawings. In some alternative implementations, the functions indicated in a block may not occur in the order shown in the drawings. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions or implements a combination of dedicated hardware and computer instructions.

[0153] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit these implementations. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code—it should be understood that software and hardware can be designed to implement systems and / or methods based on those described herein.

[0154] No element, action, or instruction used herein should be construed as essential or necessary unless explicitly stated otherwise. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “one” or similar language is used where only one item is intended. Moreover, as used herein, the terms “have,” “possess,” “contain,” etc., are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the word “based on” is intended to mean “at least partially based on.”

[0155] Descriptions of various aspects and implementations have been presented for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed implementations. Although combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each listed dependent claim may be directly subordinated to only one claim, the disclosure of possible implementations includes combinations of each dependent claim with every other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described implementations. The terminology used herein has been chosen to best illustrate the principles of the implementations, their practical application, or technical improvements beyond those found in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for streaming video data, characterized in that, The method includes: Receive a 360° video stream, wherein the 360° video stream includes multiple overlay parameters, the multiple overlay parameters indicating whether the user terminal allows two or more overlay objects to be overlaid on the 360° video stream; Display the 360° video stream; When the multiple overlay parameter is a first value, it is determined that the user terminal allows the 360° video stream to be overlaid with two or more overlay objects, and the 360° video stream having the two or more overlay objects is displayed; and When the multiple overlay parameter is the second value, it is determined that the user terminal does not allow the 360° video stream to be overlaid with two or more overlay objects, and the 360° video stream with zero overlay objects or one overlay object is displayed. Wherein, the first value is different from the second value, and Wherein, the one or more overlay objects include one or more of the following: another video stream or image; The multiple overlay parameters are proposed in the Session Description Protocol (SDP) of the 360° video stream; The 360° video stream includes a flag that signals when a user is allowed to share the sender's stream information with other senders. When the flag indicates that a user is allowed to share the sender's stream information with other senders, the sender's stream information is shared by the user terminal with other senders. When the flag indicates that a user is not allowed to share the sender's stream information with other senders, the user terminal does not share the sender's stream information with other senders.

2. The method according to claim 1, characterized in that, The 360° video stream includes overlap parameters that indicate whether it is allowed to overlap one of the two or more overlap objects with a first overlap object. The method further includes: When the overlap parameter indicates that one of the two or more overlap objects is allowed to overlap with the first overlap object, the user terminal displays the 360° video stream having the two or more overlap objects, wherein the first overlap object overlaps with one of the two or more overlap objects.

3. The method according to claim 1, characterized in that, The 360° video stream includes content-based attributes that allow the use of the two or more overlay objects.

4. The method according to claim 1, characterized in that, The method further includes: The user terminal receives a request from an electronic device or server, wherein the request is for one or more users in a remote conference to share media stream attributes; and The user terminal sends information about the media stream attributes that the user terminal is authorized to share to the electronic device or the server.

5. The method according to claim 4, characterized in that, Information regarding the media stream attributes that the user is willing to share includes an identifier (ID) corresponding to the sender ID.

6. The method according to claim 5, characterized in that, The method further includes: A unique overlay ID is assigned by the electronic device or the server by combining at least one sender ID with at least one overlay ID.

7. The method according to claim 1, characterized in that, The method further includes negotiating the multiple overlay parameters during the initial proposal-response negotiation or during the session.

8. The method according to claim 2, characterized in that, The method further includes negotiating the overlapping parameters during the initial proposal-response negotiation or during the session.

9. The method according to claim 2, characterized in that, The proposed overlap parameters are described in the Session Description Protocol (SDP) of the 360° video stream.

10. The method according to claim 3, characterized in that, The method further includes negotiating the content-based attributes during initial proposal-response negotiation or during a session.

11. The method according to claim 3, characterized in that, The content-based attributes are proposed in the Session Description Protocol (SDP) of the 360° video stream.

12. The method according to claim 1, characterized in that, The method further includes: transmitting from the user terminal to an electronic device or server whether another user is authorized to share streaming information shared by the user terminal with other electronic devices.

13. The method according to claim 4, characterized in that, The method further includes: the user terminal transmitting to the electronic device or the server whether another user is authorized to share streaming information shared by the user terminal with other electronic devices.

14. The method according to claim 12, characterized in that, The method further includes: the electronic device or the server sharing with the other electronic device at least one additional overlay object other than any one or more overlay objects generated by the electronic device or the server, wherein the at least one additional overlay object includes only overlay objects in which the sender of the corresponding overlay video has authorized the sharing of the overlay object.

15. A computer device, characterized in that, The computer device includes: One or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and execute the method according to any one of claims 1 to 14 as instructed by the computer program code.

16. A user terminal for streaming video data, the user terminal comprising: Communication interface; monitor; A receiving unit is configured to receive a 360° video stream via the communication interface, wherein the 360° video stream includes multiple overlay parameters, the multiple overlay parameters indicating whether the user terminal allows two or more overlay objects to be overlaid on the 360° video stream; Display unit, the display unit being configured to control the display to display a received 360° video stream; A multi-overlay authorization unit is configured to perform the following operations: when the multi-overlay parameter is a first value, determining that the user terminal allows the 360° video stream to be overlaid with two or more overlay objects, and controlling the display to display the 360° video stream having the two or more overlay objects; and An unauthorized multiple overlay unit is configured to perform the following operations: when the multiple overlay parameter is a second value, determine that the user terminal does not allow the 360° video stream to be overlaid with two or more overlay objects, and control the display to display the 360° video stream having zero overlay objects or one overlay object. Wherein, the first value is different from the second value, and Wherein, the one or more overlay objects include one or more of the following: another video stream or image; The multiple overlay parameters are proposed in the Session Description Protocol (SDP) of the 360° video stream; The 360° video stream includes a flag that signals when a user is allowed to share the sender's stream information with other senders. When the flag indicates that a user is allowed to share the sender's stream information with other senders, the sender's stream information is shared by the user terminal with other senders. When the flag indicates that a user is not allowed to share the sender's stream information with other senders, the user terminal does not share the sender's stream information with other senders.

17. The user terminal according to claim 16, wherein The 360° video stream includes overlap parameters, which indicate whether it is allowed to overlap one of the two or more overlap objects with a first overlap object. The user terminal also includes: An overlay authorization unit is configured to perform the following operations: when the overlay parameters indicate that one of the two or more overlay objects is allowed to overlap with the first overlay object, control the display to display the 360° video stream having the two or more overlay objects, wherein the first overlay object overlaps with one of the two or more overlay objects.

18. A non-transitory computer-readable medium having a computer program stored thereon for streaming video data, characterized in that, The computer program is configured to cause one or more computer processors to perform the method according to any one of claims 1 to 14.

Images