Immersive Media Interoperability

Abstract

Aspects of the disclosure provide methods and apparatuses for audio processing. In some examples, the apparatus for media processing includes a processing circuit. The processing circuit receives first three-degree-of-freedom (3DoF) information associated with first media content of a scene in a media application. The first 3DoF information includes a first rotational orientation for describing the first media content on a first sphere centered on a user of the media application. The processing circuit determines that a rendering platform for rendering the first media content is a six-degree-of-freedom (6DoF) platform and calculates first spatial location information of the first media content based on the first rotational orientation and a first parameter of the first sphere. The first spatial location information is used in the first 6DoF information associated with the first media content to render the first media content on the 6DoF platform.

Description

[Technical field]

[0001] [Incorporated by reference] This application claims the benefit of priority to U.S. Patent Application No. 17 / 892,987 entitled "IMMERSIVE MEDIA INTEROPERABILITY," filed August 22, 2022, which claims the benefit of priority to U.S. Provisional Application No. 63 / 260,509 entitled "Immersive Media Interoperability," filed August 23, 2021. The disclosures of the prior applications are hereby incorporated by reference in their entireties.

[0002] [Technical field] This disclosure describes embodiments generally relating to media processing. [Background technology]

[0003] The background art description provided herein is intended to generally provide a context for the disclosure. To the extent that it is described in this background art section, the works of the currently named inventors, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are not admitted expressly or implicitly as prior art to the present disclosure.

[0004] In virtual reality or augmented reality applications, video, audio, or other stimuli in the application's virtual scene are perceived as in the real world to give the user the sensation of being in the application's virtual world. In some examples, the user's physical movements in the real world are perceived as having matching movements in the application's virtual scene. Furthermore, the user can interact with the virtual scene using media that is perceived as realistic and matches the user's experience in the real world. Summary of the Invention

[0005] Aspects of the disclosure provide a method and apparatus for media processing. In some examples, the apparatus for media processing includes a processing circuit. The processing circuit receives first three degree of freedom (3DoF) information associated with a first media content of a scene in a media application. The first 3DoF information includes a first rotational orientation for describing the first media content on a first sphere centered on a user of the media application. The processing circuit determines that a rendering platform for rendering the first media content is a six degree of freedom (6DoF) platform and calculates first spatial location information of the first media content based on the first rotational orientation and a first parameter of the first sphere. The first spatial location information is used in the first 6DoF information associated with the first media content to render the first media content on the 6DoF platform.

[0006] In some embodiments, the processing circuit determines a first parameter of the first sphere according to a predefined parameter. In some examples, the processing circuit sets a center position of the first sphere to be a predefined position and sets a radius of the first sphere to be a predefined positive value. In one example, the processing circuit sets a center position of the first sphere to be (0,0,0) in three-dimensional (3D) space and sets a radius of the first sphere to be 1.

[0007] In some embodiments, the apparatus is a media client device and the processing circuit receives the first parameter of the first sphere from a media server device. In one example, the processing circuit receives the first parameter of the first sphere from a supplemental enhancement information (SEI) message that is part of a bitstream carrying the first media content. In another example, the processing circuit receives the first parameter of the first sphere from a metadata packet at a system layer. The metadata packet at the system layer is separate from the bitstream carrying the first media content.

[0008] In some examples, the processing circuit receives second 3DoF information associated with second media content of a scene in the media application. The second 3DoF information includes a second rotation orientation for describing the second media content on a second sphere that shares a center location with the first sphere. The processing circuit calculates second spatial location information of the second media content based on the second rotation orientation and a second parameter of the second sphere. The second spatial location information is used in second 6DoF information associated with the second media content to render the second media content on a 6DoF platform.

[0009] In one example, the first sphere and the second sphere have the same radius. In another example, the first sphere and the second sphere have different radii.

[0010] In one example, the apparatus is a media client device and the processing circuitry renders the first media content according to the first 6DoF information including the first spatial location information. In another example, the apparatus is a media server device and the processing circuitry provides the first media content associated with the first 6DoF information in a bitstream to the media client device.

[0011] Aspects of the disclosure further provide a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computer, cause the computer to perform a method of media processing. [Brief description of the drawings]

[0012] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] 1A-1C show diagrams illustrating environments using six degrees of freedom (6DoF) and three degrees of freedom (3DoF) in some examples. [Figure 1B]1A-1C show diagrams illustrating environments using six degrees of freedom (6DoF) and three degrees of freedom (3DoF) in some examples. [Figure 1C] 1A-1C show diagrams illustrating environments using six degrees of freedom (6DoF) and three degrees of freedom (3DoF) in some examples. [Diagram 2] 1 illustrates a block diagram of a media system according to one embodiment of the disclosure. [Diagram 3] 1 shows a flowchart outlining a process according to some embodiments of the disclosure. [Figure 4] 1 is a schematic illustration of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Aspects of the disclosure provide techniques for immersive media interoperability. With immersive technology, immersive media attempts to create or mimic a physical world through digital simulation. According to one aspect of the disclosure, devices for immersive media applications may support different degrees of freedom, for example, six degrees of freedom (6DoF), three degrees of freedom (3DoF), etc. A device that supports 6DoF may be referred to as a 6DoF platform including hardware and software that supports 6DoF. A device that supports 3DoF may be referred to as a 3DoF platform including hardware and software that supports 3DoF. Some media content is generated to interact with a user on a 6DoF platform and is referred to as 6DoF media content, and some media content is generated to interact with a user on a 3DoF platform and is referred to as 3DoF media content. The present disclosure provides techniques for tailoring immersive media content to improve immersive media interoperability. Specifically, the techniques can be applied to 3DoF media content for use on a 6DoF platform.

[0014] According to some aspects of the disclosure, some technologies attempt to create or mimic a physical world through a digital simulation called immersive media. Immersive media processing can be implemented according to immersive media standards such as the Moving Picture Expert Group Immersive (MPEG-I) suite of standards, which include "immersive audio," "immersive video," and "systems support." Immersive media standards can support VR or AR presentations in which a user can navigate and interact with an environment using six degrees of freedom (6DoF). 6DoF refers to the freedom of movement of a rigid body in three-dimensional space. Body movement can include translational and rotational motion.

[0015] FIG. 1A shows a diagram illustrating an environment (100A) using six degrees of freedom (6DoF) in some examples. The 6DoF of a user's (101A) movement can include translational and rotational movements. For example, translational movements can be represented as the spatial navigation of the user (101A) using the position (a, b, c) of the user (101A) in three-dimensional space, where a change in a is a movement along the X-axis (e.g., forward and backward), a change in b is a movement along the Y-axis (e.g., left and right), and a change in c is a movement along the Z-axis (e.g., up and down). Rotational movements can be represented as the orientation of the user using the user's head orientation (α1, β1, γ1), where α1 is the angle of rotation around the Z-axis, also called the yaw angle, β1 is the angle of rotation around the Y-axis, also called the pitch angle, and γ1 is the angle of rotation around the X-axis, also called the roll angle. As shown in FIG. 1A, the user's rotation orientation (α1, β1, γ1) is centered around the spatial position (a, b, c) of the user (101A).

[0016] Although Figure 1A shows 6DoF relative to the user, a similar definition can be applied to other rigid bodies, for example audio sources (e.g., speakers) and virtual objects in the environment.

[0017] FIG. 1B shows a diagram illustrating an environment (100B) using six degrees of freedom (6DoF) in some examples. The environment (100B) includes a user (101B) and a speaker (102B). The 6DoF of the user (101B) is defined in the same manner as in the environment (100A). For example, the 6DoF of the user (101B) includes the spatial position (a, b, c) of the user (101B) in a three-dimensional space and the rotation orientation (α1, β1, γ1) of the user (101B). The rotation orientation (α1, β1, γ1) is defined with the spatial position (a, b, c) of the user (101B) as the center.

[0018] The 6DoF of the speaker's motion can include the spatial position (x,y,z) of the speaker (102B) and the rotational orientation (α2,β2,γ2) of the speaker (102B). The rotational orientation (α2,β2,γ2) is defined with the center at the spatial position (x,y,z) of the speaker (102B).

[0019] Although Figure 1B shows 6DoF relative to a speaker, a similar definition can be applied to other rigid bodies.

[0020] In some use cases, when an immersive media device supports 6DoF, a user can interact with 6DoF media content with the support of the immersive media device. For example, the user's 6DoF information can be detected by the immersive media device, and the immersive media device can render audio and video of the 6DoF media content based on the user's 6DoF information.

[0021] Some immersive media devices, such as 3DoF platforms, may not support 6DoF and may only allow people to interact with media content (e.g., 3DoF media content) using three degrees of freedom (3DoF). In some examples of 3DoF media content, 3DoF may refer to orientation information associated with a user and media. For example, orientation information of media in a 3DoF setting includes a revolution orientation angle of the media present on a sphere where the user is the center of the sphere. The media may be visual, audio (audio channels, audio objects, etc.), or in other formats. Orientation information in a 3DoF setting may be referred to as revolution orientation (θ,φ,ψ), where θ is the rotation angle around the user's Z axis, φ is the rotation angle around the user's Y axis, and ψ is the rotation angle around the user's X axis.

[0022] FIG. 1C shows a diagram illustrating an environment (100C) using three degrees of freedom (6DoF) in some examples. The environment (100C) includes a user (101C) and a speaker (102C). The speaker (102C) is treated as a point (111) on a sphere (110). The center of the sphere (110) is the user (101C). The 3DoF of the speaker (102C) is defined as the rotational orientation (θ, φ, ψ) of the point (111) on the sphere (110).

[0023] In some applications, 3DoF media content may need to be rendered on a 6DoF platform. This disclosure provides a method for adjusting 3DoF media content to 6DoF media content to be rendered on a 6DoF platform.

[0024] 2 illustrates a block diagram of a media system 200 according to one embodiment of the disclosure. The media system 200 can be used in a variety of applications, such as immersive media applications, augmented reality (AR) applications, virtual reality applications, video game applications, sports game animation applications, video conferencing and telepresence applications, and media streaming applications.

[0025] The media system (200) includes a media server device (210) and multiple media client devices, such as media client devices (260A) and (260B) shown in FIG. 2, which may be connected by a network (not shown). In one example, the media server device (210) may include one or more devices with audio coding and video coding capabilities. In one example, the media server device (210) includes a single computing device, such as a desktop computer, a laptop computer, a server computer, a tablet computer, etc. In another example, the media server device (210) includes a data center, a server farm, etc. The media server device (210) may receive video and audio content and compress the video and audio content into one or more encoded bitstreams according to an appropriate media coding standard. The encoded bitstreams may be distributed to the media client devices (260A) and (260B) over the network.

[0026] The media client devices (e.g., media client devices 260A and 260B) each include one or more devices with video coding and audio coding capabilities for media applications. In one example, each of the media client devices includes a computing device such as a desktop computer, a laptop computer, a server computer, a tablet computer, a wearable computing device, a head mounted display (HMD) device, etc. The media client devices can decode the encoded bitstream according to an appropriate media coding standard. The decoded video and audio content can be used for media playback.

[0027] The media server device (210) may be implemented using any suitable technology. In the example of Figure 2, the media server device (210) includes a processing circuit (230) and an interface circuit (211) coupled together.

[0028] The processing circuitry (230) may include any suitable processing circuitry, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), application specific integrated circuits, etc. The processing circuitry (230) may be configured to include various encoders, such as an audio encoder, a video encoder, etc. In one example, one or more CPUs and / or GPUs may execute software that functions as an audio encoder or a video encoder. In another example, the audio encoder or the video encoder may be implemented using an application specific integrated circuit.

[0029] The interface circuitry (211) can interface the media server device (210) with a network. The interface circuitry (211) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuitry (211) can transmit signals that carry the encoded bitstream over the network to other devices, such as media client device (260A), media client device (260B), etc. The interface circuitry (211) can receive signals from media client devices, such as media client devices (260A) and (260B).

[0030] The network is suitably coupled to the media server device (210) and the media client devices (e.g., media client devices (260A) and (260B)) via wired and / or wireless connections, such as Ethernet connections, fiber optic connections, WiFi connections, cellular network connections, etc. The network may include network server devices, storage devices, network devices, etc. The components of the network are suitably coupled together via wired and / or wireless connections.

[0031] The media client devices (e.g., media client devices 260A and 260B) are each configured to decode the coded bitstream. In one example, each media client device can perform video decoding to reconstruct a sequence of video frames that can be displayed, and can perform audio decoding to generate an audio signal for playback.

[0032] Media client devices such as media client devices 260A and 260B may be implemented using any suitable technology. In the example of Figure 2, media client device 260A is shown as user equipment that can be used by user A, but is not limited to a head mounted display (HMD) with earphones, and media client device 260B is shown as user equipment that can be used by user B, but is not limited to an HMD with earphones.

[0033] In FIG. 2, the media client device (260A) includes an interface circuit (261A) and a processing circuit (270A) coupled together as shown in FIG. 2, and the media client device (260B) includes an interface circuit (261B) and a processing circuit (270B) coupled together as shown in FIG. 2.

[0034] The interface circuit (261A) can interface the media client device (260A) with a network. The interface circuit (261A) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuit (261A) can receive signals carrying data, such as signals carrying encoded bitstreams, from the network.

[0035] The processing circuitry (270A) may include any suitable processing circuitry, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuitry (270A) may be configured to include various components, such as a DoF converter (271A), a renderer (272A), a video decoder (not shown), and an audio decoder (not shown).

[0036] In some examples, the audio decoder can decode the audio content in the encoded bitstream by selecting a decoding tool appropriate for the manner in which the audio content was encoded, and the video decoder can decode the video content in the encoded bitstream by selecting a decoding tool appropriate for the manner in which the video content was encoded. The DoF converter (271A) is configured to adjust the DoF information in the decoded media content to be compatible with the media client device (260A). In one example, the hardware of the media client device (260A) supports 6DoF and the received media content includes 3DoF media content, and then the DoF converter (271A) can convert the 3DoF media content into 6DoF media content for playback on the media client device (260A).

[0037] Additionally, the renderer (272A) can generate a final digital product suitable for the media client device (260A) from the audio and video content decoded from the encoded bitstream. Note that the processing circuitry (270A) can include other suitable components (not shown), such as a mixer, post-processing circuitry, etc., for further media processing.

[0038] Similarly, the interface circuit (261B) can interface the media client device (260B) with a network. The interface circuit (261B) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuit (261B) can receive signals carrying data, such as signals carrying encoded bitstreams, from the network.

[0039] The processing circuit (270B) may include any suitable processing circuit, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuit (270B) may be configured to include various components, such as a DoF converter (271B), a renderer (272B), a video decoder, an audio decoder, etc.

[0040] In some examples, the audio decoder can decode the audio content in the encoded bitstream by selecting a decoding tool appropriate for the manner in which the audio content was encoded, and the video decoder can decode the video content in the encoded bitstream by selecting a decoding tool appropriate for the manner in which the video content was encoded. The DoF converter (271B) is configured to adjust the DoF information in the received media content to be compatible with the media client device (260B). In one example, the hardware of the media client device (260B) supports 6DoF and the received media content includes 3DoF media content, and then the DoF converter (271B) can convert the 3DoF media content into 6DoF media content for playback on the media client device (260B).

[0041] Additionally, the renderer (272B) can generate a final digital product suitable for the media client device (260B) from the audio content decoded from the encoded bitstream. Note that the processing circuitry (270B) can include other suitable components (not shown), such as a mixer, post-processing circuitry, etc., for further audio processing.

[0042] In some examples, the media server device (210) can perform the DoF conversion. In one example, the media server device (210) includes a DoF converter (231). In one example, the media server device (210) can receive hardware information from a media client device, such as a media client device (260A) and a media client device (260B). For example, the media server device (210) receives hardware information from the media client device (260A), where the hardware information indicates that the media client device (260A) supports 6DoF media content and is in a low battery state, and then the DoF converter (231) can convert the 3DoF media content to 6DoF media content. The 6DoF media content can be appropriately encoded and transmitted to the media client device (260A). In another example, the media server device (210) receives hardware information from the media client device (260B), where the hardware information indicates that the media client device (260B) supports 6DoF media content and the media client device (260) lacks the processing capabilities (e.g., does not have a DoF converter), and the DoF converter (231) can then convert the 3DoF media content into 6DoF media content, which can then be appropriately encoded and transmitted to the media client device (260B).

[0043] According to some aspects of the disclosure, 3DoF media content can describe user orientation information along with orientation information of media in a scene. The media content can be visual, audio (audio channels, audio objects, etc.), or in other formats. To convert the 3DoF media content into 6DoF media content, in some examples, user space position information and spatial position information of each media in a scene are appropriately determined.

[0044] It is noted that the 3DoF information of the 3DoF media content describes the rotational orientation of the media content based on a sphere where the user is the center of the sphere and the media content is at the surface location of the sphere. According to one aspect of the disclosure, the 3DoF information of the media content can be converted to the 6DoF information of the media content using the parameters of the sphere, such as the center location of the sphere, the radius of the sphere, etc. The parameters of the sphere can be predefined and available on the server side or the client side, or can be sent from the server side to the client side.

[0045] In some embodiments, a set of predefined parameters for the sphere is available at the server side and the client side (e.g., the media server device and the media client device in FIG. 2), and the 6DoF media content can be generated corresponding to the 3DoF media content based on the set of predefined parameters. In some examples, the user space position and the distance between the user and the media content position (e.g., the radius of the sphere where the media content position is on the sphere and the center of the sphere is the user position) are the predefined parameters.

[0046] In some examples, the user space position is predefined as being at (0,0,0), the origin of the three-dimensional space. For 3DoF media content with rotational orientation (yaw θ, pitch φ, roll ψ), the spatial position (x,y,z) can be calculated based on the rotational orientation.

[0047] In one example, the user space position is set to (0,0,0), and the distance between the position of the media content and the position of the user is set to 1. For example, the position of the media content is on a unit sphere centered on the user. Thus, the spatial position (x,y,z) of the media content can be calculated, for example, according to Equations (1), (2), and (3).

number

[0048] Therefore, 6DoF can be used to represent the movement of a user and media content in an environment. For example, the spatial position information of a user in 6DoF can be represented by (0,0,0), and the rotation orientation of the user in 6DoF can be represented by (θ,φ,ψ). The spatial position information of a media content in 6DoF can be represented by (x,y,z) calculated according to Equation (1), Equation (2), and Equation (3), and the rotation orientation of the media content can be set by any suitable value such as (θ,φ,ψ), (2π-θ,π-φ,π-ψ), (0,0,0), etc.

[0049] In another example, the spatial position of the user is set to (0,0,0), and the distance between the spatial position of the media content and the spatial position of the user is r, where r is a positive number. Therefore, the spatial position of the media content is on a sphere with a radius of r, and the spatial position of the user is the center of the sphere. The spatial position (x,y,z) of the media content can be calculated according to Equation (4), Equation (5), and Equation (6).

number

[0050] Therefore, 6DoF can be used to represent the movement of a user and media content in an environment. For example, the spatial position information of a user in 6DoF can be represented by (0,0,0), and the rotation orientation of the user in 6DoF can be represented by (θ,φ,ψ). The spatial position information of a media content in 6DoF can be represented by (x,y,z) calculated according to Equation (4), Equation (5), and Equation (6), and the rotation orientation of the media content can be set by any suitable value such as (θ,φ,ψ), (2π-θ,π-φ,π-ψ), (0,0,0), etc.

[0051] In another example, the spatial position of the user is set as (a, b, c), and the distance between the spatial position of the media content and the spatial position of the user is set to be r, where r is a positive number. The spatial position (x, y, z) of the media content can be calculated according to Equation (7), Equation (8), and Equation (9).

number

[0052] Therefore, 6DoF can be used to represent the movement of a user and media content in an environment. For example, the spatial position information of a user in 6DoF can be represented by (a, b, c), and the rotation orientation of the user in 6DoF can be represented by (θ, φ, ψ). The spatial position information of a media content in 6DoF can be represented by (x, y, z) calculated according to Equation (7), Equation (8), and Equation (9), and the rotation orientation of the media content can be set by any suitable value such as (θ, φ, ψ), (2π-θ, π-φ, π-ψ), (0, 0, 0), etc.

[0053] In some embodiments, the spatial locations of the different media content lie on the same sphere with the user's spatial location at its center, and therefore all media content shares the same radius r.

[0054] In some embodiments, each media content has its own radius r. The radii of the media content may be different. Thus, the spatial locations of the media content may lie on different spheres centered on the user location.

[0055] According to one aspect of the disclosure, the spherical parameters (e.g., the location of the center of the sphere, the radius of the sphere) in addition to the 3DOF media content may be delivered to the client side via metadata or SEI messages, either as part of the content bitstream or separately. With the combined information of both the 3DOF source and the spherical parameters, the 6DOF platform can render the corresponding content.

[0056] In one embodiment, the parameters of the sphere, including the user space position (center of the sphere), the distance between the user and the position of the media content (radius of the sphere), are signaled as part of the bitstream carrying the 3DoF media content, such as by using supplemental enhanced information (SEI) messages. The SEI messages do not affect the decoding process of the immersive 3DoF media content from the content bitstream. The SEI messages can assist in rendering 6DoF effects (on a 6DoF platform) with the provided 3DoF media content.

[0057] In another embodiment, the spherical parameters, including the user space position (center of the sphere), the distance between the user and the position of the media content (radius of the sphere), are not delivered as part of the bitstream carrying the 3DoF media content. In some examples, the spherical parameters are transmitted as metadata, such as via a system layer packet. The metadata does not affect the normal decoding process of the immersive 3DoF content from the bitstream. The metadata can assist in rendering a 6DoF effect (e.g., on a 6DoF platform) with the provided 3DoF media content.

[0058] 3 shows a flow chart outlining a process (300) according to one embodiment of the disclosure. The process (300) may be performed by a media processing device, such as a DoF converter (231) in a media server device (210), a DoF converter (271A) in a media client device (260A), or a DoF converter (271B) in a media client device (260B). In some embodiments, the process (300) is implemented with software instructions, such that the processing circuitry performs the process (300) when the processing circuitry executes the software instructions. The process starts at (S301) and proceeds to (S310).

[0059] At (S310), first three degree of freedom (3DoF) information associated with first media content of a scene in a media application is received. The first 3DoF information includes a first rotational orientation for describing the first media content on a first sphere. In one example, the first sphere is centered on a user of the media application.

[0060] At (S320), the media processing device determines that a rendering platform for rendering the media content is a 6DoF platform. In one example, the media processing device is a media client device that supports 6DoF, and the media processing device can make the determination based on a configuration of the media client device. In another example, the media processing device is a media server device, and the media server device can receive a signal from the media client device. The signal indicates that the media client device supports 6DoF.

[0061] At (S330), first spatial location information of the first media content is determined based on the first rotational orientation and the first parameter of the first sphere. The first spatial location information is used in first 6DoF information associated with the first media content to render the first media content on a 6DoF platform, in one example.

[0062] In some embodiments, the first parameter of the first sphere is determined according to a predefined parameter known to the media server device and the media client device. In some examples, the center position of the first sphere is set to be a predefined position, the radius of the first sphere is set to be a predefined positive value, and the first spatial position can be determined according to, for example, Equation (7), Equation (8), and Equation (9). In one example, the center position of the first sphere is set to be (0,0,0) in three-dimensional (3D) space, and the radius of the first sphere is set to be 1. The first spatial position information can then be determined according to, for example, Equation (1), Equation (2), and Equation (3).

[0063] In some embodiments, the media processing device is a media client device, and the media client device receives the first parameters of the first sphere from the media server device. In some examples, the media client device receives the first parameters of the first sphere from a supplemental enhancement information (SEI) message that is part of a bitstream carrying the first media content. The first media content can be decoded from the bitstream without using the SEI message. The SEI message provides information for rendering a 6DoF effect with the first media content that is 3DoF.

[0064] In some examples, the media client device receives the first parameters of the first sphere from a packet of metadata at a system layer, the packet being separate from a bitstream carrying the first media content. The first media content can be decoded from the bitstream without using the metadata. The metadata provides information for rendering a 6DoF effect with the first media content, which is 3DoF.

[0065] In some examples, second 3DoF information associated with a second media content of a scene in a media application is received. The second 3DoF information includes a second rotation orientation to describe the second media content on a second sphere that shares a center location with the first sphere. Second spatial location information of the second media content is then calculated based on the second rotation orientation and a second parameter of the second sphere. The second spatial location information is used in second 6DoF information associated with the second media content to render the second media content on a 6DoF platform. In one example, the first sphere and the second sphere have the same radius. In another example, the first sphere and the second sphere have different radii.

[0066] In some examples, the media processing device is a media client device, and the media client device renders the first media content according to the first six DoF information that includes the first spatial location information.

[0067] In some examples, the media processing device is a media server device, and the media server device provides the first media content associated with the first 6DoF information in a bitstream to the media client device.

[0068] Thereafter, the process proceeds to (S399) and ends.

[0069] The process (300) may be adapted as appropriate. Steps in the process (300) may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0070] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 4 illustrates a computer system (400) suitable for implementing certain embodiments of the disclosed subject matter.

[0071] The computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to produce code containing instructions that may be executed directly, or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0072] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.

[0073] 4 for computer system (400) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (400).

[0074] The computer system (400) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Additionally, the human interface devices may be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0075] The input human interface devices may include one or more of a keyboard (401), a mouse (402), a trackpad (403), a touch screen (410), a data glove (not shown), a joystick (405), a microphone (406), a scanner (407), and a camera (408) (only one of each is shown).

[0076] The computer system (400) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (410), data gloves (not shown), joystick (405), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (409), headphones (not shown)), visual output devices (e.g., screens (410), such as CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or output in more than three dimensions through means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0077] The computer system (400) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (420) with media (421) such as CDs / DVDs, thumb drives (422), removable hard drives or solid state drives (423), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0078] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0079] The computer system (400) may also include an interface (454) to one or more communication networks (455). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular, and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CANBus. Certain networks typically require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (449) (e.g., a USB port of the computer system (400)). Others are typically integrated into the core of the computer system (400) by attachment to a system bus as described below (e.g., an Ethernet interface in a PC computer system, or a cellular network interface in a smartphone computer system). Using any of these networks, the computer system (400) may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), transmit only unidirectional (e.g., a CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of the networks and network interfaces as described above.

[0080] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (440) of the computer system (400).

[0081] The cores (440) may include one or more central processing units (CPUs) (441), graphics processing units (GPUs) (442), specialized programmable processing units (443) in the form of field programmable gate areas (FPGAs), hardware accelerators (444) for specific tasks, graphics adapters (450), etc. These devices may be connected through a system bus (448), along with read only memory (ROM) (445), random access memory (446), and internal mass storage devices (447), such as internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus (448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached either directly to the core's system bus (448) or through a peripheral bus (449). In one example, a screen (410) may be connected to the graphics adapter (450). Peripheral bus architectures include PCI, USB, etc.

[0082] The CPU (441), GPU (442), FPGA (443), and accelerator (444) can execute certain instructions that, in combination, may constitute the aforementioned computer code. The computer code may be stored in ROM (445) or RAM (446). Transient data may also be stored in RAM (446), while persistent data may be stored, for example, in internal mass storage (447). Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU (441), GPU (442), mass storage (447), ROM (445), RAM (446), etc.

[0083] The computer-readable medium can bear computer code for performing various computer-implemented operations. The media and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0084] By way of example and not limitation, a computer system having the architecture (400), and in particular the core (440), can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage devices as introduced above, as well as media associated with the core's (440) specific storage devices that are non-transitory in nature, such as the core's internal mass storage device (447) or ROM (445). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (440). The computer-readable media can include one or more memory devices or chips according to specific needs. The software can cause the core (440), and in particular the processor therein (including CPU, GPU, FPGA, etc.) to perform certain operations or certain portions of certain operations described herein, including defining data structures stored in RAM (446) and modifying such data structures according to software-defined operations. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (444)), which may operate in place of or in conjunction with software to perform particular operations, or particular portions of particular operations, described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0085] While this disclosure has described several exemplary embodiments, there are modifications, permutations, and various substitute equivalents that fall within the scope of the disclosure. Thus, it will be appreciated that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method of media processing performed by a media processing device, comprising: receiving first three degrees of freedom (3DoF) information associated with first media content of a scene in a media application, the first 3DoF information including a first rotational orientation to describe the first media content on a first sphere centered on a user of the media application; determining that a rendering platform for rendering the first media content is a six degree of freedom (6DoF) platform; Based on the first turning orientation and a first parameter of the first sphere including a center location and a radius of the first sphere, [0010] where (x, y, z) represents a spatial position of the first media content, (θ, φ, ψ) represents the first rotation orientation, (a, b, c) represent the center position of the first sphere, and r represents the radius of the first sphere, and the first spatial position information is used in first 6DoF information associated with the first media content for rendering the first media content on the 6DoF platform; The method includes:

2. using predefined parameters as the first parameters of the first sphere. The method of claim 1 further comprising:

3. setting the center location of the first sphere to be a predefined location; setting the radius of the first sphere to be a predefined positive value; The method of claim 2 , further comprising:

4. setting the center position of the first sphere to be (0,0,0) in three-dimensional (3D) space; setting a radius of the first sphere to be 1; The method of claim 3 further comprising:

5. the media processing device is a media client device, the method comprising: receiving the first parameters of the first sphere from a media server device; The method of claim 1 further comprising:

6. The method comprises: receiving the first parameters of the first sphere from a supplemental enhancement information (SEI) message that is part of a bitstream carrying the first media content; The method of claim 5 further comprising:

7. The method comprises: receiving the first parameters of the first sphere from a metadata packet at a system layer, the metadata packet at the system layer being separate from a bitstream carrying the first media content; The method of claim 5 further comprising:

8. receiving second 3DoF information associated with second media content of the scene in the media application, the second 3DoF information including a second rotational orientation to describe the second media content on a second sphere co-centered with the first sphere; calculating second spatial location information of the second media content based on the second rotational orientation and second parameters of the second sphere, the second spatial location information being used in second 6DoF information associated with the second media content for rendering the second media content on the 6DoF platform; The method of claim 1 further comprising:

9. The method of claim 8 , wherein the first sphere and the second sphere have the same radius.

10. The method of claim 8 , wherein the first sphere and the second sphere have different radii.

11. the media processing device is a media client device, the method comprising: Rendering the first media content according to the first 6DoF information including the first spatial location information. The method of claim 1 further comprising:

12. the media processing device is a media server device, the method comprising: providing the first media content associated with the first 6DoF information in a bitstream to a media client device. The method of claim 1 further comprising:

13. A media processing device, comprising:

13. Apparatus comprising a processing circuit, the processing circuit being configured to perform a method according to any one of claims 1 to 12.

Images