Immersive Media Compatibility

Abstract

Aspects of the present disclosure provide methods and devices for audio processing. In some examples, the device for media processing includes a processing circuit. The processing circuit receives first six degrees of freedom (6DoF) information associated with media content of a scene of a media application. The first 6DoF information includes a first spatial location and a first rotational orientation for rotation around the first spatial location. The processing circuit determines that a rendering platform for rendering the media content is a three degrees of freedom (3DoF) platform and calculates, according to at least the first spatial location, an orientation of rotational movement of the media content on a sphere centered at a location other than the first spatial location. The orientation of the rotational movement is the 3DoF information associated with the media content for rendering on the 3DoF platform.

Description

[Technical field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 17 / 892,976, entitled "IMMERSIVE MEDIA COMPATIBILITY," filed August 22, 2022, which claims the benefit of priority to U.S. Provisional Application No. 63 / 260,508, entitled "IMMERSIVE MEDIA COMPATIBILITY," filed August 23, 2021. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments that generally relate to media processing. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is not admitted, expressly or impliedly, as prior art to the present disclosure, to the extent that it is described in this background section, as well as aspects of the description that may not be admitted as prior art at the time of filing.

[0004] In a virtual reality or augmented reality application, video, audio, or other stimuli in the application's virtual scene are perceived as real-world to give the user a sense of presence in the application's virtual world. In some examples, the user's physical movements in the real world are perceived to be consistent with movements in the application's virtual scene. Furthermore, the user can interact with the virtual scene using media that is perceived as realistic and consistent with the user's experience in the real world. Summary of the Invention

[0005] Aspects of the present disclosure provide a method and apparatus for media processing. In some examples, the apparatus for media processing includes a processing circuit. The processing circuit receives first six degrees of freedom (6DoF) information associated with media content of a scene of a media application. The first 6DoF information includes a first spatial location and a first rotation orientation for rotation around the first spatial location. The processing circuit determines that a rendering platform for rendering the media content is a three degrees of freedom (3DoF) platform and calculates a revolution orientation of the media content on a sphere centered other than the first spatial location according to at least the first spatial location. The rotation orientation is the 3DoF information associated with the media content for rendering on the 3DoF platform.

[0006] In some embodiments, the first spatial location is described in a three-dimensional (3D) Cartesian coordinate system by three coordinate values ​​respectively associated with three axes of the 3D Cartesian coordinate system.

[0007] In some examples, the processing circuitry receives second 6DoF information associated with a user of the media application. The second 6DoF information includes a second spatial location and a second rotational orientation for rotation about the second spatial location. The processing circuitry calculates an orientation of rotational movement of the media content on a sphere centered at the second spatial location according to the first spatial location and the second spatial location.

[0008] In one example, the processing circuitry calculates a distance between the first spatial location and the second spatial location, and the distance is used to adjust a gain of the audio of the media content.

[0009] In some examples, the processing circuitry determines a rotation matrix for rotating on the sphere from a starting point on the sphere to a first spatial location on the sphere and calculates an orientation of the rotational motion based on the rotation matrix.

[0010] In some examples, the processing circuitry calculates a distance between the first spatial location and the second spatial location and determines an end point on a unit sphere that corresponds to the first spatial location. The unit sphere is centered at the second spatial location and has a unit radius. The processing circuitry determines a rotation matrix for rotating on the unit sphere from the start point to the end point and calculates an orientation of the rotational motion based on the rotation matrix.

[0011] In some embodiments, the first spatial location is described in a three-dimensional (3D) polar coordinate system by a radial distance, an azimuth angle, and a polar angle, and in some examples, the processing circuitry converts the azimuth angle into a yaw angle of orientation of the rotational motion and converts the polar angle into pitch and roll angles of orientation of the rotational motion.

[0012] In some examples, the media processing device is a media client device and the processing circuitry renders the media content according to an orientation of the rotational motion.

[0013] In some examples, the media processing device is a media server device, and the processing circuitry then provides media content associated with the rotational orientation from the media server device to the media client device.

[0014] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for media processing. [Brief description of the drawings]

[0015] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] FIG. 1 illustrates an environment using six degrees of freedom (6DoF) and three degrees of freedom (3DoF) in some examples. [Figure 1B] FIG. 1 illustrates an environment using six degrees of freedom (6DoF) and three degrees of freedom (3DoF) in some examples. [Figure 1C]FIG. 1 illustrates an environment using six degrees of freedom (6DoF) and three degrees of freedom (3DoF) in some examples. [Diagram 2] FIG. 1 is a block diagram of a media system according to one embodiment of the present disclosure. [Diagram 3] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Aspects of the present disclosure provide techniques for improving compatibility of immersive media. With immersive technology, immersive media seeks to form or mimic the physical world through digital simulation. According to one aspect of the present disclosure, a device for an immersive media application can support different degrees of freedom, such as six degrees of freedom (6DoF) and three degrees of freedom (3DoF). A device that supports 6DoF can be referred to as a 6DoF platform including hardware and software that supports 6DoF. A device that supports 3DoF can be referred to as a 3DoF platform including hardware and software that supports 3DoF. Some media content is generated to interact with a user on a 6DoF platform and is referred to as 6DoF media content. Some media content is generated to interact with a user on a 3DoF platform and is referred to as 3DoF media content. The present disclosure provides techniques for adjusting immersive media content to improve compatibility of immersive media. Specifically, these techniques can be applied to 6DoF media content for use on a 3DoF platform.

[0017] According to some aspects of the present disclosure, some techniques attempt to create or mimic a physical world through a digital simulation called immersive media. Immersive media processing can be implemented according to immersive media standards such as the Moving Picture Expert Group Immersive (MPEG-I) suite of standards, which includes "immersive audio", "immersive video", and "system support". Immersive media standards can support VR or AR presentations in which users can navigate and interact with the environment using six degrees of freedom (6DoF). 6DoF refers to the degrees of freedom of rigid body movement in three-dimensional space. The rigid body movement includes translational and rotational motion.

[0018] FIG. 1A illustrates an environment (100A) that uses six degrees of freedom (6DoF) in some examples. The user's 6DoF motion includes translational motion and rotational motion. For example, translational motion can be represented as a user's spatial navigation using the user's position (a, b, c) in a three-dimensional space. A change in a is a motion along the X-axis (e.g., forward / backward), a change in b is a motion along the Y-axis (e.g., left / right), and a change in c is a motion along the Z-axis (e.g., up / down). Rotational motion can be represented as a user's orientation using the user's head orientation (α1, β1, γ1). Here, α1 is the rotation angle around the Z-axis, also called the yaw angle; β1 is the rotation angle around the Y-axis, also called the pitch angle; and γ1 is the rotation angle around the X-axis, also called the roll angle. As shown in FIG. 1A, the user's rotational orientation (α1, β1, γ1) is centered on the user's spatial position (a, b, c).

[0019] While Figure 1A shows 6DoF for the user, similar definitions can be applied to other rigid bodies. For example, 6DoF can be similarly defined for sound sources such as speakers, virtual objects, etc. in the environment (100A).

[0020] FIG. 1B shows a diagram illustrating an environment (100B) using six degrees of freedom (6DoF) in some examples. The environment (100B) includes a user and a speaker. The user's 6DoF is defined similarly to the environment (100A). For example, the user's 6DoF includes the user's spatial position (a, b, c) in a three-dimensional space and the user's rotational orientation (α1, β1, γ1). The rotational orientation (α1, β1, γ1) is defined with the user's spatial position (a, b, c) as the center.

[0021] The 6DoF movement of a speaker may include the spatial position (x1, y1, z1) of the speaker and the rotational orientation (α2, β2, γ2) of the speaker, where the rotational orientation (α2, β2, γ2) is defined around the spatial position (x1, y1, z1) of the speaker.

[0022] While FIG. 1B shows 6DoF for a loudspeaker, similar provisions can be applied to other rigid bodies.

[0023] In some usage scenarios, when an immersive media device supports 6DoF, a user can interact with 6DoF media content with the support of the immersive media device. For example, the user's 6DoF information can be detected by the immersive media device, and the immersive media device can render audio and video of the 6DoF media content based on the user's 6DoF information.

[0024] Some immersive media devices, such as 3DoF platforms, may not support 6DoF and may only allow people to interact with media content (e.g., 3DoF media content) using three degrees of freedom (3DoF). In some examples of 3DoF media content, 3DoF may refer to orientation information associated with the user and the media. For example, orientation information for media in a 3DoF setting includes the orientation angle of the rotational motion of the media on a sphere with the user at the center of the sphere. The media can be visual, audio (audio channels, audio objects, etc.), or in other forms. Orientation information in a 3DoF setting may be referred to as the orientation of the rotational motion (θ,φ,ψ), where θ is the angle of revolution around the user's Z axis, φ is the angle of revolution around the user's Y axis, and ψ is the angle of revolution around the user's X axis.

[0025] FIG. 1C shows a diagram illustrating an environment (100C) that uses three degrees of freedom (6DoF) in some examples. The environment (100C) includes a user and a speaker. The speaker is treated as a point (111) on a sphere (110). The center of the sphere (110) is the user. The 3DoF of the speaker is defined as the orientation (θ, φ, ψ) of the rotational motion of the point (111) on the sphere (110).

[0026] In some applications, 6DoF media content may need to be rendered on a 3DoF platform. This disclosure provides techniques for adjusting 6DoF media content to 3DoF media content for rendering on a 3DoF platform.

[0027] 2 illustrates a block diagram of a media system 200 according to one embodiment of the present disclosure, which may be used in a variety of use applications, such as immersive media applications, augmented reality (AR) applications, virtual reality applications, video game applications, sports game animation applications, teleconferencing and telepresence applications, and media streaming applications.

[0028] The media system (200) includes a media server device (210) and multiple media client devices, such as media client devices (260A) and (260B) shown in FIG. 2, which may be connected by a network (not shown). In one example, the media server device (210) may include one or more devices with audio coding and video coding capabilities. In one example, the media server device (210) includes a single computing device, such as a desktop computer, a laptop computer, a server computer, and a tablet computer. In another example, the media server device (210) may include a data center, a server farm, and the like. The media server device (210) may receive video and audio content and compress the video and audio content into one or more encoded bitstreams according to an appropriate media coding standard. The encoded bitstreams may be distributed to the media client devices (260A) and (260B) over the network.

[0029] Each of the media client devices (e.g., media client devices 260A and 260B) includes one or more devices having video coding and audio coding capabilities for media applications. In one example, each of the media client devices includes a computing device such as a desktop computer, a laptop computer, a server computer, a tablet computer, and a wearable computing device and a head mounted display (HMD) device. The media client devices can decode the encoded bitstream according to an appropriate media coding standard. The decoded video and audio content can be used for media playback.

[0030] The media server device (210) may be implemented using any suitable technology. In the example of Figure 2, the media server device (210) includes a processing circuit (230) and an interface circuit (211) coupled together.

[0031] The processing circuitry (230) may include any suitable processing circuitry, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), and application specific integrated circuits. The processing circuitry (230) may be configured to include various encoders, such as an audio encoder and a video encoder. In one example, one or more CPUs and / or GPUs may execute software to function as an audio encoder or a video encoder. In another example, the audio encoder or the video encoder may be implemented using an application specific integrated circuit.

[0032] The interface circuit (211) can interface the media server device (210) with a network. The interface circuit (211) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuit (211) can transmit signals carrying encoded bitstreams over the network to other devices, such as media client device (260A) and media client device (260B). The interface circuit (211) can receive signals from media client devices, such as media client devices (260A) and (260B).

[0033] The network is suitably coupled to the media server device 210 and the media client devices (e.g., media client devices 260A and 260B) via wired and / or wireless connections, such as Ethernet connections, fiber optic connections, WiFi connections, and cellular network connections. The network may include network server devices, storage devices, and network devices. The components of the network are suitably coupled via wired and / or wireless connections.

[0034] Each media client device (e.g., media client devices 260A and 260B) is configured to decode the encoded bitstream. In one example, each media client device can perform video decoding to reconstruct a sequence of video frames that can be displayed, and can perform audio decoding to generate an audio signal for playback.

[0035] Media client devices such as media client devices 260A and 260B may be implemented using any suitable technology. In the example of FIG. 2, media client device 260A is shown, but is not limited to a head mounted display (HMD) with earphones as user equipment that can be used by user A, and media client device 260B is shown, but is not limited to a HMD with earphones as user equipment that can be used by user B.

[0036] In Figure 2, media client device (260A) includes an interface circuit (261A) and a processing circuit (270A) coupled together as shown in Figure 2. Media client device (260B) includes an interface circuit (261B) and a processing circuit (270B) coupled together as shown in Figure 2.

[0037] The interface circuit (261A) can interface the media client device (260A) with a network. The interface circuit (261A) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuit (261A) can receive signals carrying data, such as signals carrying encoded bitstreams, from the network.

[0038] The processing circuitry (270A) may include any suitable processing circuitry, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuitry (270A) may be configured to include various components, such as a DoF converter (271A), a renderer (272A), a video decoder (not shown), and an audio decoder (not shown).

[0039] In some examples, the audio decoder can decode the audio content in the encoded bitstream by selecting a decoding tool suitable for the scheme with which the audio content was encoded, and the video decoder can decode the video content in the encoded bitstream by selecting a decoding tool suitable for the scheme with which the video content was encoded. The DoF converter (271A) is configured to adjust the DoF information in the decoded media content to be compatible with the media client device (260A). In one example, if the hardware of the media client device (260A) only supports 3DoF and the received media content includes 6DoF media content, the DoF converter (271A) can convert the 6DoF media content into 3DoF media content and play it on the media client device (260A).

[0040] Additionally, the renderer (272A) can generate a final digital product suitable for the media client device (260A) from the audio and video content decoded from the encoded bitstream. It should be noted that the processing circuitry (270A) can include other suitable components (not shown), such as a mixer and post-processing circuitry, for further media processing.

[0041] Similarly, the interface circuit (261B) can interface the media client device (260B) with a network. The interface circuit (261B) can include a receiving portion that receives signals from the network and a transmitting portion that transmits signals to the network. For example, the interface circuit (261B) can receive signals carrying data, such as signals carrying encoded bitstreams, from the network.

[0042] The processing circuitry (270B) may include any suitable processing circuitry, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuitry (270B) may be configured to include various components, such as a DoF converter (271B), a renderer (272B), a video decoder, and an audio decoder.

[0043] In some examples, the audio decoder can decode the audio content in the encoded bitstream by selecting a decoding tool suitable for the scheme with which the audio content was encoded, and the video decoder can decode the video content in the encoded bitstream by selecting a decoding tool suitable for the scheme with which the video content was encoded. The DoF converter (271B) is configured to adjust the DoF information in the received media content to be compatible with the media client device (260B). In one example, if the hardware of the media client device (260B) only supports 3DoF and the received media content includes 6DoF media content, the DoF converter (271B) can convert the 6DoF media content into 3DoF media content and play it on the media client device (260B).

[0044] Additionally, the renderer (272B) can generate a final digital product suitable for the media client device (260B) from the audio content decoded from the encoded bitstream. It should be noted that the processing circuitry (270B) can include other suitable components (not shown), such as a mixer and post-processing circuitry, for further audio processing.

[0045] In some examples, the media server device (210) can perform the DoF conversion. In one example, the media server device (210) includes a DoF converter (231). In one example, the media server device (210) can receive hardware information from a media client device, such as a media client device (260A) and a media client device (260B). For example, if the media server device (210) receives hardware information from the media client device (260A) and the hardware information indicates that the media client device (260A) supports only 3DoF media content and is in a low battery state, the DoF converter (231) can convert the 6DoF media content to 3DoF media content. The 3DoF media content can be appropriately encoded and transmitted to the media client device (260A). In another example, when the media server device (210) receives hardware information from the media client device (260B) and the hardware information indicates that the media client device (260B) only supports 3DoF media content and the media client device (260B) lacks processing capabilities (e.g., does not have a DoF converter), the DoF converter (231) can convert the 6DoF media content into 3DoF media content, which can be appropriately encoded and transmitted to the media client device (260B).

[0046] According to one aspect of the disclosure, 6DoF media content can describe media (e.g., sound sources, audio channels, audio objects, visual objects, virtual objects, etc.) using 6DoF, which can include spatial location information of the media (in 3D space) and rotation information (with respect to the media's axes). In some examples, the spatial location information can be provided as 3D Cartesian coordinates (x,y,z), where x is the distance along the X axis, y is the distance along the Y axis, and z is the distance along the Z axis. In some examples, the spatial location information can be provided in a polar coordinate system as (r,ω,φ), where r is the radial distance, ω is the azimuth angle, and φ is the polar angle. Rotation information of the media in 6DoF can be provided as a rotation orientation (α,β,γ), where α is the rotation angle of the media about the Z axis, β is the rotation angle of the media about the Y axis, and γ is the rotation angle of the media about the X axis.

[0047] According to another aspect of the present disclosure, 3DoF media content can describe media (e.g., sound sources, virtual objects, etc.) using rotational orientation information with respect to a user. In one example, the rotational orientation information for each media in the 3DoF media content includes a rotational orientation (θ, φ, ψ) for the media residing on a sphere, with the user residing at the center of the sphere. θ is the rotation angle around the Z axis of the center of the sphere, φ is the rotation angle around the Y axis of the center of the sphere, and ψ is the rotation angle around the X axis of the center of the sphere. The media may be visual, audio (audio channels, audio objects, etc.), or in other formats.

[0048] According to some aspects of the present disclosure, in order to adjust 6DoF media content to 3DoF media content, the orientation of the rotational movement of each media on the sphere (with the user at the center of the sphere) can be calculated.

[0049] In some examples, the spatial position information of 6DoF can be described using a 3D Cartesian coordinate system. The rotation angle can be calculated based on the spatial position information of the user and the spatial position information of the media. In one example, the spatial position information of the user in 6DoF is denoted by (a, b, c), and the spatial position information of the media in 6DoF is denoted by (x1, y1, z1). To convert to 3DoF media content, the user is assumed to be at the center of a sphere and the media is assumed to be at a surface position of the sphere. The radius r of the sphere can be calculated according to formula (1).

number

[0050] In some examples, in the case of audio media (audio channels, audio objects, etc.), the media client device may adjust the gain (volume) of the audio media in response to changes in position. In one example, the gain may be adjusted inversely proportional to the radius r.

[0051] Furthermore, the spatial position (x1,y1,z1) of the media can be normalized to (x2,y2,z2), which is a position on a unit sphere (with radius equal to 1) centered at (0,0,0), e.g., according to equations (2), (3), and (4).

number

[0052] Further, in some embodiments, a rotation matrix can be determined based on the normalized position (x2, y2, z2) and the starting point on the unit sphere, and a rotation angle (θ, φ, ψ) can be determined based on the rotation matrix.

[0053] In some examples, the starting point may be set to (0,0,1), and the rotation matrix R may be determined to satisfy equation (5).

number

[0054] In one example, a rotation matrix R is determined and has the form of equation (6):

number

[0055] Next, the orientation (θ, φ, ψ) of the rotational movement relative to the starting point can be determined based on the rotation matrix, in one example according to equations (7), (8), and (9).

number

[0056] Note that in the above example, the point (0,0,1) is selected as the starting point for determining the rotation matrix R. The selection of the starting point is exemplary. In some examples, other points such as (1,0,0), (0,1,0), etc. may be selected as the starting point of the rotation matrix. The derivation of the rotation angles (θ,φ,ψ) may be adjusted appropriately.

[0057] According to another aspect of the present disclosure, the spatial location information of media in 6DoF media content can be described using a 3D polar coordinate system (r,ω,φ), where r is the radial distance, ω is the azimuth angle, and φ is the polar angle. Each of the azimuth angle ω and the polar angle φ can have a range of 360 degrees. According to one aspect of the present disclosure, the azimuth angle ω and the polar angle can be mapped to rotation angles (yaw θ, pitch φ, roll ψ) in 3DoF. In one example, the azimuth angle ω is mapped to yaw θ, and the polar angle φ can be appropriately converted to pitch φ and roll ψ. In this example, the yaw θ has a range of 360 degrees, and the pitch φ and roll ψ each have a range of 180 degrees. In some examples, when converting media content from 6DoF to 3DoF, the radial information of the spatial location information can be removed and the angular information can be mapped to rotation angles (yaw θ, pitch φ, roll ψ) to fit the 3DoF media content format.

[0058] In one example, the user may be at the origin of a 3D polar coordinate system, with the yaw angle θ being equal to the azimuth angle ω.

[0059] 3 shows a flow chart outlining a process (300) according to one embodiment of the present disclosure. The process (300) may be performed by a media processing device, such as a DoF converter (231) in a media server device (210), a DoF converter (271A) in a media client device (260A), and a DoF converter (271B) in a media client device (260B). In some embodiments, the process (300) is implemented in software instructions, such that the processing circuitry executes the software instructions to perform the process (300). Processing begins at (S301) and proceeds to (S310).

[0060] At (S310), first 6DoF information associated with media content of a scene of a media application is received, the first 6DoF information including a first spatial location and a first rotational orientation about the first spatial location (center of rotation).

[0061] At (S320), the media processing device determines that the rendering platform for rendering the media content is a 3DoF platform. In one example, the media processing device is a media client device that supports only 3DoF, and the media processing device can make the determination based on the configuration of the media client device. In another example, the media processing device is a media server device, and the media server device can receive a signal from the media client device. The signal indicates that the media client device supports only 3DoF.

[0062] In (S330), a rotational movement orientation of the media content on a sphere centered at a point other than the first spatial position is calculated according to at least the first spatial position, the rotational movement orientation being 3DoF information associated with the media content for rendering on a 3DoF platform.

[0063] In some embodiments, the first spatial location is described in a three-dimensional (3D) Cartesian coordinate system by three coordinate values ​​for three axes. In some examples, the media processing device receives second 6DoF information associated with a user of a media application. The second 6DoF information includes the second spatial location and a second rotational orientation about the second spatial location. The media processing device can calculate an orientation of the rotational movement of the media content on a sphere centered on the second spatial location according to the first spatial location and the second spatial location.

[0064] In some examples, the media processing device calculates a distance between the first spatial location and the second spatial location, the distance being used to adjust a gain of the audio of the media content.

[0065] In some examples, the media processing device determines a rotation matrix for rotating on the sphere from a starting point on the sphere to a first spatial position on the sphere, and calculates an orientation of the rotational motion based on the rotation matrix.

[0066] In some examples, the media processing device calculates a distance between the first spatial location and the second spatial location, and determines an end point on a unit sphere corresponding to the first spatial location. The unit sphere is centered on the second spatial location and has a unit radius. The media processing device then determines a rotation matrix for rotating on the unit sphere from the start point to the end point, and calculates a direction of the rotational motion based on the rotation matrix. In one example, the start point is (0,0,1), and the direction of the rotational motion can be calculated according to Equation (7), Equation (8), and Equation (9).

[0067] In some embodiments, the first spatial location is described in a three-dimensional (3D) polar coordinate system by a radial distance, an azimuth angle, and a polar angle. In one example, the media processing device converts the azimuth angle into a yaw angle of rotational orientation and the polar angle into pitch and roll angles of rotational orientation.

[0068] In some examples, the media processing device is a media client device, and the media client device is capable of rendering media content according to the orientation of the rotational movement.

[0069] In some examples, the media processing device is a media server device, and the media processing device can provide media content associated with an orientation of rotational motion from the media server device to a media client device using a 3DoF platform for rendering the media content.

[0070] Next, the process proceeds to (S399) and ends.

[0071] The process (300) may be adapted as appropriate. Steps of the process (300) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0072] The techniques described above may be implemented as computer software using computer readable instructions and may be physically stored on one or more computer readable media. For example, Figure 4 illustrates a computer system (400) suitable for implementing certain embodiments of the disclosed subject matter.

[0073] The computer software may be coded using any suitable machine code or computer language that, following assembly, compilation, linking, or similar mechanisms, produces code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), or the like, either directly or through interpretation, as microcode.

[0074] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0075] 4 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (400).

[0076] The computer system (400) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly related to human input consciousness, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still camera), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0077] The input human interface devices may include one or more of a keyboard (401), a mouse (402), a trackpad (403), a touch screen (410), a data glove (not shown), a joystick (405), a microphone (406), a scanner (407), and a camera (408) (only one of each is shown).

[0078] The computer system (400) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (410), data gloves (not shown), or joystick (405), but may also be haptic feedback devices that do not function as input devices), audio output devices (speakers (409), headphones (not shown), etc.), visual output devices (such as screens (410), including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touch screen input capability, each with or without haptic feedback capability, some of which may output two-dimensional visual output or three or more dimensional output via means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0079] The computer system (400) may include human accessible storage and associated media such as optical media, including CD / DVD ROM / RW (420) with CDs / DVDs or similar media (421), thumb drives (422), removable hard drives or solid state drives (423), conventional magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices (not shown) such as security dongles, etc.

[0080] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0081] The computer system (400) may also include an interface (454) to one or more communication networks (455). The networks may be, for example, wireless, wired, optical, etc. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, and LTE, TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Certain networks generally require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (449) (e.g., a USB port on the computer system (400)). Others are generally integrated into the core of the computer system (400) by connecting to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system), as described below. Using any of these networks, the computer system (400) can communicate with other entities. Such communications may be either unidirectional, receive only (e.g., TV broadcast), unidirectional transmit only (e.g., from the CANbus to a particular CANbus device), or bidirectional (e.g., to other computer systems using local or wide area digital networks). As noted above, each of these networks and network interfaces may use specific protocols and protocol stacks.

[0082] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (440) of the computer system (400).

[0083] The core (440) may include one or more central processing units (CPUs) (441), graphics processing units (GPUs) (442), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (443), hardware accelerators for specific tasks (444), and graphics adapters (450). These devices may be connected via a system bus (448), along with read only memory (ROM) (445), random access memory (446), and internal mass storage devices (447) such as internal hard drives or SSDs that are not accessible to the user. In some computer systems, the system bus (448) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the core's system bus (448) or through a peripheral bus (449). In one example, a screen (410) may be connected to the graphics adapter (450). Peripheral bus architectures include PCI, USB, and the like.

[0084] The CPU (441), GPU (442), FPGA (443), and accelerator (444) may execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in ROM (445) or RAM (446). Transient data may also be stored in RAM (446), while permanent data may be stored, for example, in internal mass storage (447). The use of cache memory allows for fast storage and retrieval in any memory device, the use of which may be closely related to one or more of the CPU (441), GPU (442), mass storage (447), ROM (445), and RAM (446), etc.

[0085] The computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0086] By way of example and not limitation, a computer system having the architecture (400), and in particular the core (440), can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be mass storage accessible to a user as introduced above, and media associated with a particular storage of the core (440) of a non-transitory nature, such as the core internal mass storage (447) or ROM (445). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (440). The computer-readable media can include one or more memory devices or chips, depending on the particular need. The software can cause the core (440), and in particular the processor therein (including CPU, GPU, FPGA, etc.) to perform a particular process or a particular part of a particular process described herein, including defining data structures stored in RAM (446) and modifying such data structures according to the process defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic that is hardwired or embedded in circuitry (e.g., accelerator (444)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0087] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents which fall within the scope of this disclosure. Thus, it should be appreciated that those skilled in the art can devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. 1. A method for processing media on a media processing device, the method comprising: receiving first six degrees of freedom (6DoF) information associated with media content of a scene of a media application, the first 6DoF information including a first spatial position and a first rotational orientation about the first spatial position; receiving second 6DoF information associated with a user of the media application, the second 6DoF information including a second spatial location and a second rotational orientation about the second spatial location; determining that a rendering platform for rendering the media content is a three degree of freedom (3DoF) platform; and calculating, according to the first spatial position and the second spatial position, a rotational movement orientation of the media content on a sphere centered on the second spatial position, the rotational movement orientation being 3DoF information associated with the media content for rendering on the 3DoF platform. method.

2. The method of claim 1 , wherein the first spatial location is described in a three-dimensional (3D) Cartesian coordinate system by three coordinate values ​​respectively associated with three axes of the 3D Cartesian coordinate system.

3. Calculating an orientation of the rotational movement of the media content on the sphere centered at the second spatial location comprises:

10. The method of claim 1, further comprising: calculating a distance between the first spatial location and the second spatial location; and using the distance to adjust a gain of audio of the media content.

4. Calculating an orientation of the rotational movement of the media content on the sphere centered at the second spatial location comprises: determining a rotation matrix for rotating on the sphere from a starting point on the sphere to the first spatial location on the sphere; The method of claim 1 , further comprising: calculating an orientation of the rotational motion based on the rotation matrix.

5. Calculating an orientation of the rotational movement of the media content on the sphere centered at the second spatial location comprises: calculating a distance between the first spatial location and the second spatial location; determining an end point on a unit sphere corresponding to the first spatial location, the unit sphere being centered at the second spatial location and having a unit radius; determining a rotation matrix for rotating on the unit sphere from a start point to the end point; The method of claim 1 , further comprising: calculating an orientation of the rotational motion based on the rotation matrix.

6. The method of claim 1 , wherein the first spatial location is described in a three-dimensional (3D) polar coordinate system by a radial distance, an azimuth angle, and a polar angle.

7. Calculating the orientation of the rotational movement of the media content on the sphere comprises: converting the azimuth angle into a yaw angle in the direction of the rotational motion; and converting the polar angle into pitch and roll angles of orientation of the rotational motion.

8. The media processing device is a media client device, the method comprising: The method of claim 1 , further comprising rendering the media content according to an orientation of the rotational movement.

9. The media processing device is a media server device, the method comprising: The method of claim 1 , further comprising providing the media content associated with the rotational orientation from the media server device to a media client device.

10. 1. A media processing device including a processing circuit, comprising: The processing circuitry includes:

10. Configured to perform the method according to any one of claims 1 to 9. device.

11. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:

10. Carrying out the method according to any one of claims 1 to 9, A non-transitory computer-readable storage medium.

Images