Method, system and storage medium for notifying audio mixing gain by signaling
By using SDP and RTP header extensions for audio mixing gain signaling in immersive remote meetings, the problem of uneven mixing of 360-degree background audio streams and superimposed audio streams is solved, improving audio output quality and user experience.
Patent Information
- Application Number
- CN202280004013.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-11
- Filing Date
- 2022-03-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-03-14
AI Technical Summary
In immersive remote meetings, it is difficult for the prior art to effectively mix and manage 360-degree background audio streams and superimposed audio streams, resulting in uneven audio output and affecting the user experience.
Signaling of audio mixing gains for 360-degree media streams and superimposed audio streams ensures real-time updates and reasonable allocation of audio mixing gains by using Session Description Protocol (SDP) and Real-Time Transfer Protocol (RTP) header extensions.
A balanced mix of 360-degree background audio streams and superimposed audio streams is achieved, improving the audio output quality in remote meetings, and improving the user experience and dynamic management capabilities of audio signals.
Smart Images

Figure CN115516836B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 160,630 filed on March 12, 2021 and U.S. Application No. 17 / 692,840 filed on March 11, 2022, the disclosures of which are incorporated herein by reference in their entirety. Technical Field
[0003] Embodiments of the present disclosure relate to signaled audio mixing gains for immersive teleconferencing and telepresence for remote terminals (ITT4RT), and more specifically to defined Session Description Protocol (SDP) attributes and Real-time Transport Protocol (RTP) header extensions for signaling audio mixing gains for mixing 360-degree background and 360-degree overlay audio streams together based on different usage scenarios. Background Art
[0004] When using omnidirectional media streaming, only a portion of the content corresponding to the user's viewport is presented while using a head mounted display (HMD), thereby providing the user with a realistic view of the media stream.
[0005] Figure 1 The figure shows a related technical scenario (scenario 1) for an immersive teleconference call, where the call is organized between room A (101), user B (102) and user C (103). Figure 1 As shown in , Room A (101) represents a conference room with an omnidirectional / 360-degree camera (104), and User B (102) and User C (103) are remote participants using an HMD and a remote participant using a mobile device, respectively. In this case, participants User B (102) and User C (103) send their viewport orientations to Room A (101), and Room A (101) in turn sends the viewport-related streams to User B (102) and User C (103).
[0006] Figure 2A An extended scenario (scenario 2) is shown, which includes multiple conference rooms (2a01, 2a02, 2a03, 2a04). User B (2a06) uses an HMD to view a video stream from a 360-degree camera (104), and user C (2a07) uses a mobile device to view the video stream. User B (2a06) and user C (2a07) send their viewport orientation to at least one of the conference rooms (2a01, 2a02, 2a03, 2a04), and at least one of the conference rooms (2a01, 2a02, 2a03, 2a04) in turn sends the viewport-related stream to user B (2a06) and user C (2a07).
[0007] like Figure 2B As shown in, in another example scenario (Scenario 3), when a call is established using MRF / MCU (2b05), the Media Resource Function (MRF) and the Media Control Unit (MCU) are multimedia servers that provide media-related functions for bridging terminals in a multi-party conference call. Conference rooms can send their respective videos to the MRF / MCU (2b05). These videos are viewport-independent videos, that is, the entire 360-degree video is sent to the media server (i.e., MRF / MCU) regardless of the user viewport that streams the specific video. The media server receives the viewport orientation of the user (User B (2b06) and User C (2b07)) and sends the viewport-related streams to the users accordingly.
[0008] Further to scenario 3, the remote user may choose to watch one of the available 360-degree videos from the conference room (2a01-2a04, 2b01-2b04). In this case, the user sends information about the video he wants to stream and its viewport orientation to the conference room or MRF / MCU (2b05). The user may also trigger a switch from one room to another based on active speakers. The media server may pause receiving video streams from any conference room without active users.
[0009] ISO 23090-2 defines overlay as "a block of visual media presented over an omnidirectional video or image item or over a viewport". When any participant in conference room A is sharing any presentation, in addition to being displayed in conference room A, the presentation is also broadcast as a stream to other users (conference rooms 2a02-2a04, 2b02-2b04, user B (2b06) and / or user C (2b07)). This stream can be overlaid on top of the 360 video. Additionally, overlay can also be used for 2D streams. The default audio mixing gains for different audio streams are 360 video (a0) and overlay video (a1, a2, ..., a3) respectively. N ) of the audio gain (r0, r1, ..., r N ), and the audio output is equal to r0*a0+r1*a1+......+r n *a n , where r0+r1+...+r N = 1. The receiver or MRF / MCU mixes the audio source in proportion to its mixing gain. Summary of the invention
[0010] One or more example embodiments of the present disclosure provide a system and method for signaling audio mixing gains.
[0011] According to an embodiment, a method for notifying audio mixing gain by signaling is provided herein. The method may include: receiving an input audio stream from a 360-degree media stream, the input audio stream including an audio mixing gain, wherein the audio mixing gain includes a first audio mixing gain from the input audio stream and a second audio mixing gain from a superimposed audio stream; using a session description protocol SDP to declare a real-time transport protocol RTP header extension of the input audio stream, the RTP header extension including an element identifier, an element length, and an amplitude of the audio mixing gain corresponding to the element; and based on a single attribute of the SDP, using the RTP header extension to signal the audio mixing gain.
[0012] According to an embodiment, a system for notifying audio mixing gain by signaling is provided. The system may include at least one memory storing instructions and at least one processor, wherein the at least one processor is configured to execute instructions to: receive an input audio stream from a 360-degree media stream, the input audio stream including an audio mixing gain, wherein the audio mixing gain includes a first audio mixing gain from the input audio stream and a second audio mixing gain from a superimposed audio stream; use a session description protocol SDP to declare a real-time transport protocol RTP header extension of the input audio stream, the RTP header extension including an element identifier, an element length, and an amplitude of the audio mixing gain corresponding to the element; and based on a single attribute of the SDP, use the RTP header extension to signal the audio mixing gain.
[0013] According to an embodiment, a non-volatile computer-readable medium for signaling an audio mixing gain is provided herein. The non-volatile computer-readable medium can be connected to at least one processor and can be configured to store instructions, which when executed cause: at least one processor to receive an input audio stream from a 360-degree media stream, the input audio stream including an audio mixing gain, wherein the audio mixing gain includes a first audio mixing gain from the input audio stream and a second audio mixing gain from a superimposed audio stream; use a session description protocol SDP to declare a real-time transport protocol RTP header extension of the input audio stream, the RTP header extension including an element identifier, an element length, and an amplitude of the audio mixing gain corresponding to the element; and based on a single attribute of the SDP, use the RTP header extension to signal the audio mixing gain.
[0014] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be achieved by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The above and other aspects, features, and aspects of embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the following drawings.
[0016] Figure 1 is a schematic diagram of the ecosystem for immersive remote conferencing.
[0017] Figure 2A It is a schematic diagram of a multi-party multi-conference room remote conference.
[0018] Figure 2B This is a schematic diagram of a multi-party, multi-conference room remote conference using MRF / MCU.
[0019] Figure 3 is a simplified block diagram of a communication system in accordance with one or more embodiments.
[0020] Figure 4 is a simplified example illustration of a streaming environment in accordance with one or more embodiments.
[0021] Figure 5A is a diagram illustrating audio mixing gains using a one-byte RTP header extension according to one or more embodiments.
[0022] Figure 5B is a diagram illustrating audio mixing gains using a two-byte RTP header extension according to one or more embodiments.
[0023] Figure 6 is a flow chart of a method for SDP signaling audio mixing gains in a teleconference using an RTP header extension according to one or more embodiments.
[0024] Figure 7 is a schematic diagram of a computer system according to one or more embodiments. DETAILED DESCRIPTION
[0025] The present disclosure relates to methods and systems for signaling audio gains from overlay and 360 degree streams in a teleconference with SDP. The present disclosure further relates to signaling audio mixing gains of overlay and 360 degree streams in a teleconference for audio mixing via RTP header extension.
[0026] like Figure 2A and Figure 2BAs shown in , multiple conference rooms with omnidirectional cameras are in a remote conference, and a user selects a video / audio stream from one of the conference rooms (2a01, 2a02, 2a03, 2a04) to be displayed as an immersive stream. Any additional audio or video stream used with the 360-degree immersive stream is sent as an overlay (i.e., as a separate stream). As soon as the terminal device receives multiple audio streams, it decodes them and mixes them for presentation to the user. The sending conference room provides the mixing gain levels of all different audio streams. The sending conference room can also update the mixing gain levels of different audio streams during the remote conference session. Audio mixing gains can be defined for each audio stream. Therefore, it will be necessary to apply the method as detailed in the embodiments of the present disclosure, which uses a single header extension to send / receive all audio gains (r0, r1, ..., r N ) and superimposed video (a1, a2, ..., a N ).
[0027] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. However, examples of implementation can be implemented in various forms, and the present disclosure should not be construed as being limited to the examples described herein. On the contrary, the examples of implementation are provided to make the technical solutions of the present disclosure more comprehensive and complete, and to fully convey the ideas of the examples of implementation to those skilled in the art. The accompanying drawings are merely examples of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the accompanying drawings represent the same or similar parts, and therefore repeated descriptions of these parts are omitted.
[0028] The proposed features discussed below may be used alone or in any order. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. Further, the embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits), or in the form of software, or in different networks and / or processor devices and / or microcontroller devices. In one example, one or more processors execute a program stored in a non-volatile computer-readable medium.
[0029] Figure 3is a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) may include at least two terminals interconnected via a network (305), namely a second terminal (302) and a third terminal (303). For unidirectional transmission of data, the third terminal (303) may encode video data at a local location for transmission to the second terminal (302) via the network (305). The second terminal (302) may receive the encoded video data of the third terminal from the network (305), decode the encoded data and display the recovered video data. Unidirectional data transmission is common in media service applications such as teleconferencing.
[0030] Figure 3 A second pair of terminals, a first terminal (301) and a fourth terminal (304), are illustrated, which are provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each of the first terminal (301) and the fourth terminal (304) can encode video data captured at a local location for transmission to the other terminal via a network (305). Each of the first terminal (301) and the fourth terminal (304) can also receive encoded video data transmitted by the other terminal, can decode and mix the encoded data, and can display the recovered video data on a local display device.
[0031] exist Figure 3 In the embodiment of the present invention, the first terminal (301), the second terminal (302), the third terminal (303) and the fourth terminal (304) can be exemplified as a server, a personal computer and a smart phone, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (250) represents any number of networks that transmit encoded video data between the first terminal (301), the second terminal (302), the third terminal (303) and the fourth terminal (304), including, for example, wired (wired) and / or wireless communication networks. The communication network (250) can exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. The mixed gain discussed in the embodiments of the present invention can be sent and received via the network (305) or the like, using the network protocol explained below.
[0032] Figure 4 An example streaming environment for applications of the disclosed subject matter is illustrated. The disclosed subject matter can be equally applied to other video-enabled applications, including, for example, immersive teleconference calls, video teleconferencing, and telepresence, among others.
[0033] The streaming environment may include one or more conference rooms (403), which may include a video source (401), such as a camera, and one or more participants of a conference (402). Figure 4 The video source (401) illustrated in the figure is, for example, a 360-degree camera that can create a video sample stream. The video sample stream can be sent to a streaming server (404) and / or stored on the streaming server (404) for future use. One or more streaming clients (405, 406) can also send their respective viewport information to the streaming server (404). Based on the viewport information, the streaming server (404) can send the viewport-related stream to the corresponding streaming client (405, 406). In another example embodiment, the streaming client (405, 406) can access the streaming server (404) to retrieve the viewport-related stream. The streaming server (404) and / or the streaming client (405, 406) can include hardware, software, or a combination thereof to support or implement various aspects of the disclosed subject matter as described in more detail below.
[0034] In an immersive teleconference call, multiple audio streams may be sent from a sender (e.g., 403) to a streaming client (e.g., 405 and / or 406). These streams may include an audio stream for a 360-degree video and one or more audio streams for superposition. The streaming client (405, 406) may include a mixing component (407a, 407b). The mixing component may decode and mix the 360-degree video and the superimposed viewport-related streams, and create an output video sample stream that may be presented on a display 408 or other presentation device such as an HDM, a speaker, a mobile device, etc. The embodiment is not limited to this configuration, and one or more conference rooms (403) may communicate with the streaming client (405, 406) via a network (e.g., network 305).
[0035] The sender can indicate the mixing gain of 360-degree audio and overlay audio by including the following media-level attribute in the SDP offer:
[0036] 3gpp_audio_mix_gain
[0037] The audio mixing gain can be described using the following Augmented Backus-Naur Form (ABNF):
[0038] a=3gpp_audio_mix_gain: <audio-mixing-gain>
[0039] When using an SDP based solution, audio mixing gain updates may be infrequent to avoid network congestion. Additionally, in the SDP offer, the sender may signal an array of mixing gains for the 360 video stream and all overlays in the media attributes of the SDP offer by using the following ABNF:
[0040] a=3gpp_audio_mix_gain:3gpp_360audio:<360_mixing_gain>
[0041] <overlay_id> :<overlay_mixing_gain>
[0042] In the previous ABNF, 3gpp_360audio is the identifier of the 360-degree audio stream, and <360_mixing_gain> represents the mixing gain value of 360audio.<overlay_id> represents a unique overlay ID used as an identifier for the overlay, and<overlay_mixing_gain> Indicates the audio mix gain of the overlay. The identifier is used to map the audio mix value to the audio stream in the media line. According to one or more embodiments, multiple overlays with unique overlay IDs can exist in the same SDP attribute.
[0043] If the sender sends an SDP offer with the attribute "3gpp_audio_mix_gain", the receiver may or may not accept the offer. If the receiver accepts the offer, the receiver MUST use the mixing gain recommended by the sender to mix the 360 audio stream and the overlaid audio. The receiver MAY change the value of the mixing gain during the session.
[0044] If the receiver chooses to use the recommended mixing gain, the sender can send updates of the audio mixing gain during the session when the mixing gain value changes. When this happens, the sender can only send the media attribute "a=3gpp_audio_mix_gain:", which maps the updated value of the mixing gain to the 360audio and overlay ID identifiers, without sending the entire media line. These identifiers can be used to map the updated audio mixing gain values to the individual audio streams.
[0045] Table 1 is an example of an SDP offer sent from a sender to a receiver (eg, a streaming client (405, 406)). In the following example, 3gpp audio and two overlay audios are mixed together.
[0046] Table 1
[0047]
[0048]
[0049] The line "a=3gpp_audio_mix_gain:3gpp_360audio:712:513:4" represents the mixing gain of the 360 audio and the overlay. "3gpp_360audio" is an identifier of the audio of the 360 audio stream with a mixing gain value of 7, "12" is an overlay ID and serves as an identifier of the overlay with a mixing gain value of "5", and "13" is an identifier of the second overlay with a mixing gain value of "4".
[0050] When the sender needs to update the mix gain of 360 audio or overlay, the sender can send a media attribute that defines "3gpp_audio_mix_gain". Below is an example of an SDP update for updating the audio mix gain, where a 360 audio stream with a mix gain value of 4 is updated.
[0051] a=3gpp_audio_mix_gain:3gpp_360audio:4 12:9 13:4
[0052] According to the above SDP, the mixing gain value of the overlay with identifier "12" is updated from value "5" to value "9". The mixing gain value of the second overlay with identifier "13" is not updated.
[0053] refer to Figure 5A and Figure 5B , signaling the audio mixing gain from the server to the client will now be described according to an embodiment.
[0054] RTP-based solutions can be used to signal the audio mixing gain from the server to the client. , The packets of the RTP audio stream can indicate in the RTP header extension the mixing gain of the audio samples to be carried in the RTP packets with the RTP extension. Figure 5A As shown in , a one-byte header format may be used for RTP extension. Elements of the header extension of the RTP extension with a one-byte header format may include ID (5a01), length L (5a02), and mixing gain (5a03).
[0055] ID (5a01) is a 4-bit ID that is the local identifier of the element. Length L (5a02) is the 4-bit length of the data bytes of the header extension element minus one and follows the one-byte header. Mixing Gain (5a03) represents the magnitude of the mixing gain for a single byte of the header extension.
[0056] like Figure 5B As shown in, the two-byte header format can also be used for RTP extension. The elements of the header extension of the RTP extension with the two-byte header format can include ID (5b01), length L (5b02) and mixing gain (5b03). The two-byte header format can also include a padding (5b04) byte with a value of 0, for example.
[0057] ID (5b01) is an 8-bit ID, which is the local identifier of the element. Length L (5b02) is an 8-bit length field, which is the length of the extended data in bytes, excluding the ID and length fields. Mixing Gain (5b03) indicates the magnitude of the mixing gain.
[0058] In the Session Description Protocol (SDP) extmap attribute, the audio mixing gain header extension is declared and mapped. The audio mixing gain header extension is declared in the SDP extmap attribute, and the audio mixing gain header extension is mapped to the uniform resource identifier (URI) of the local extension header identifier as follows:
[0059] urn: 3gpp: rtp-hdrext: audio-mixing-gai
[0060] The URI identifies and describes the header extension. In some example embodiments, the header extension may only be present in the first packet of the RTP audio stream, and may be repeated when the mixing gains need to be updated for optimization.
[0061] Figure 6 6 is a flow chart of a method 600 for signaling an audio mixing gain according to an embodiment. The method 600 may be executed in a computer system, for example.
[0062] like Figure 6 As shown in , in operation 610, method 600 includes receiving an input audio stream from a 360-degree media stream. The input audio stream includes an audio mixing gain. The input audio stream may be from a 360-degree video / audio stream in a remote conference. The audio mixing gain may include a first audio mixing gain from the input audio stream and a second audio mixing gain from the superimposed audio stream.
[0063] In operation 620, method 600 includes using SDP to declare the RTP header extension of the input audio stream. The RTP header extension includes elements, wherein each element includes an element identifier, the length of the element and the amplitude of the audio mixing gain corresponding to the element. The RTP header extension may only exist in the first packet of the input audio stream or each packet of the input audio stream. The identifier of the element includes: an identifier of the input audio stream and an identifier of the superimposed audio stream.
[0064] In operation 630, method 600 includes a single attribute based on SDP, using RTP header extension to signal audio mixing gain. The extended Backus-Naur Form (ABNF) of SDP can be used to describe audio mixing gain and a single attribute. The RTP header extension can use a one-byte RTP header extension format or a two-byte RTP header extension format to signal the audio mixing gain. In addition, the audio mixing gain can be updated. When the audio signal is updated, only the media level part of the single attribute of the specified audio mixing gain is signaled, and the RTP header extension is repeated. Method 600 can be based on a single attribute of the SDP, using the RTP header extension to signal the array of audio mixing gain from the input audio stream and from the superimposed audio stream.
[0065] Although Figure 6 An example block diagram of the method is shown, but in some implementations, the method may include Figure 6 The blocks described in the method may include additional blocks, fewer blocks, different blocks, or blocks arranged in a different manner than those depicted in the method. Additionally or alternatively, two or more of the blocks of the method may be executed in parallel.
[0066] The above-described techniques for signaling audio mixing gains for teleconferencing and telepresence may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is shown.
[0067] Computer software may be encoded using any suitable machine code or computer language that may be assembled, compiled, linked or similar mechanisms to create code comprising instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., either directly or through interpretation, microcode execution, etc.
[0068] These instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.
[0069] Figure 7 The components for computer system 700 shown in the example are exemplary in nature and are not intended to impose any limitations on the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of components should also not be interpreted as having any dependency or requirement on any one component or combination of components illustrated in the exemplary embodiment of computer system 700.
[0070] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove motions), audio input (such as voice, tapping), visual input (such as gestures), olfactory input. Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (such as voice, music, ambient sounds), pictures (such as scanned images, photographic images obtained from a still image camera), and videos (such as two-dimensional video, three-dimensional video including stereoscopic video).
[0071] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard 701 , touchpad 702 , mouse 703 , touch screen 709 , data gloves, joystick 704 , microphone 705 , camera 706 , scanner 707 .
[0072] The computer system 700 may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through touch screen 709, data gloves, or joystick 704, but there may also be tactile feedback devices that are not used as input devices), audio output devices (such as speakers 708, headphones), visual output devices (such as screens 709, including CRT screens, LCD screens, plasma screens, OLED screens, each screen with or without touch screen input capabilities, each screen with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through, for example, stereo output; virtual reality glasses, holographic displays, and smoke cans) and printers.
[0073] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 711 with CD / DVD etc. media 710, a thumb drive 712, a removable hard drive or solid state drive 713, traditional magnetic media such as tapes and floppy disks, dedicated ROM / ASIC / PLD based devices such as security dongles, and the like.
[0074] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.
[0075] The computer system 700 may also include an interface 715 to one or more communication networks 714. The network 714 may be, for example, wireless, wired, optical. The network 714 may further be local, wide area, metropolitan, vehicle-mounted, and industrial, real-time, delay-tolerant, and the like. Examples of networks 714 include local area networks (such as Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CANBus), and the like. Some networks 714 typically require an external network interface adapter (e.g., a graphics adapter 725) that is attached to some general data port or peripheral bus 716 (e.g., a USB port of the computer system 700); other systems are typically integrated into the core of the computer system 700 by attaching to a system bus as described below (e.g., an Ethernet interface is integrated into a PC computer system or a cellular network interface is integrated into a smart phone computer system). Using any of these networks 714, the computer system 700 can communicate with other entities. Such communication may be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks may be used on each of the networks and network interfaces such as those described above.
[0076] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core 717 of the computer system 700 .
[0077] The core 717 may include one or more central processing units (CPUs) 718, graphics processing units (GPUs) 719, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 720, hardware accelerators 721 for certain tasks, and the like. These devices, along with read-only memory (ROM) 723, random access memory (RAM) 724, internal mass storage devices 722 such as internal non-user accessible hard drives, SSDs, and the like, may be connected via a system bus 726. In some computer systems, the system bus 726 may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to or through the peripheral bus 716 to the core's system bus 726. The architecture of the peripheral bus includes PCI, USB, and the like.
[0078] The CPU 718, GPU 719, FPGA 720, and accelerator 721 may execute certain instructions, the combination of which may constitute the aforementioned computer code. The computer code may be stored in ROM 723 or RAM 724. Transitional data may be stored in RAM 724, while permanent data may be stored in, for example, an internal mass storage device 722. Fast storage and retrieval of any memory device may be enabled by using a cache memory, which may be closely associated with one or more CPUs 718, GPUs 719, mass storage devices 722, ROM 723, RAM 724, etc.
[0079] The computer readable medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be specially designed and constructed for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.
[0080] As an example and not by way of limitation, a computer system having architecture 700 and in particular core 717 may provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of core 717 having non-volatile properties (such as core internal mass storage devices 722 or ROM 723). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 717. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software may enable core 717 and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM 724, and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise contained in circuits (e.g., accelerator 721) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (such as integrated circuits (ICs)) storing software for execution, circuits containing logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.
[0081] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.
Claims
1. A method for signaling an audio mixing gain, characterized in that The method comprises: receiving an input audio stream from a 360-degree media stream, the input audio stream comprising an audio mixing gain; signaling the audio mixing gain in a Session Description Protocol (SDP) offer, wherein the SDP offer comprises a first audio mixing gain from the input audio stream, an identifier of the input audio stream, a second audio mixing gain from an overlay audio stream, and an identifier of the overlay audio stream; In response to the recipient accepting the SDP offer and in response to the change of the audio mixing gain, only the updated value of the audio mixing gain is signaled instead of signaling the entire media line.
2. The method according to claim 1, wherein: An extended Backus-Naur form (ABNF) of the SDP is used to describe the audio mixing gains.
3. The method according to claim 1, wherein: A one-byte Real-time Transport Protocol (RTP) header extension format is used to signal the audio mixing gain.
4. The method according to claim 1, wherein: A two-byte RTP header extension format is used to signal the audio mixing gain.
5. The method according to claim 1, wherein: The Uniform Resource Identifier (URI) in the SDP is used to declare the RTP header extension, and The RTP header extension only exists in the first packet of the input audio stream.
6. The method according to claim 1, wherein: further comprising updating the audio mixing gain, The updating frequency is less than a predetermined value.
7. The method according to claim 6, wherein: When the audio mixing gain is updated, the RTP header extension is repeated.
8. The method according to claim 1, wherein: The signaling of the audio mixing gain in the SDP proposal further comprises: An array of the audio mixing gains from the input audio stream and the overlay audio stream is signaled using an RTP header extension based on a single attribute of the SDP offer.
9. A system for signaling audio mixing gain, characterized in that The system comprises: at least one memory storing instructions; and At least one processor, the at least one processor being configured to execute the instructions to implement the method according to any one of claims 1-8.
10. A non-volatile computer-readable medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor performs the method of any one of claims 1-8.
