Immersive media data processing method and device, storage medium and electronic device

Abstract

Embodiments of the present invention provide a method for obtaining immersive media data corresponding to a user's current viewing posture, wherein a three-dimensional viewing space is used to restrict the rendering of the user's viewing content; rendering the immersive media data corresponding to the current viewing posture, wherein the obtained immersive media data is rendered when the current viewing posture is within the three-dimensional viewing space; or, when the current viewing posture has moved or is moving outside the three-dimensional viewing space, rendering the immersive media data based on processing information of the three-dimensional viewing space. This invention resolves issues that affect the rendering and presentation of visual content viewed by the user, thereby achieving the effect of facilitating the rapid, efficient, and high-quality reconstruction and rendering of visual content during the viewing process, thereby achieving an optimal immersive experience.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of communications, and in particular, to a method and device for processing immersive media data, a storage medium, and an electronic device. Background Art

[0002] Immersive media uses audio and video technologies to allow users to experience highly realistic virtual spaces, visually and auditorily, creating a sense of being there. Currently, immersive experiences primarily support two-dimensional panoramic videos. For example, users wearing a head-mounted display (HMD) can freely rotate their heads to view 360-degree videos, a three-degree-of-freedom (3DOF) immersive experience. For videos that support enhanced three-degree-of-freedom (3DOF+) and six-degree-of-freedom (6DOF), users can translate their bodies and rotate their heads to see more details, such as obscured visual content.

[0003] When a user moves their body or rotates their head, the content of the window they view changes. The display device needs to select appropriate video data from the media data based on the user's viewing posture (body position, head orientation), reconstruct and render it, and present the visual content within the current window to the user, satisfying the user's sense of immersion. The user's range of movement in space is directly related to the position, orientation, and other properties of the video data acquisition device. When the user moves beyond a certain range, the display device will be unable to reconstruct the visual content within the current window based on the acquired video data. As a result, the visual content viewed by the user will gradually fade out, become distorted, and lack immersion. To ensure the continuity and quality of the user's viewing experience, the user's three-dimensional viewing space needs to be limited. If within the three-dimensional viewing space, the user can experience immersive media normally through the rendering of the video data. If they exceed the three-dimensional viewing space, they can continue viewing and restore immersion through reconstruction and rendering of the video data, or fade out without further processing.

[0004] Currently, immersive media systems limit the scope of a user's three-dimensional viewing space to a single viewpoint with a very limited movable range. The geometric structure representing the three-dimensional viewing space is simple, such as a cube, circle, or cylinder. Since the size and shape of the three-dimensional viewing space are inevitably related to the location and number of acquisition devices, how to accurately describe the user's three-dimensional viewing space will affect the rendering and presentation of the visual content viewed by the user.

[0005] To address the above issues, the present invention proposes a method and device for processing immersive media data, a storage medium, and an electronic device, which are suitable for limiting the range of viewing positions when watching immersive media based on multi-viewpoint or point cloud data, and facilitate fast, efficient, and high-quality reconstruction and rendering of visual content during the viewing process to provide an optimal immersive experience. Summary of the Invention

[0006] Embodiments of the present invention provide a method and device for processing immersive media data, a storage medium, and an electronic device to at least solve the problems in the related art that affect the rendering and presentation of visual content viewed by a user.

[0007] According to one embodiment of the present invention, a method for processing immersive media data is provided, comprising: obtaining immersive media data corresponding to a user's current viewing posture, wherein a three-dimensional viewing space is used to limit the rendering of the user's viewing content; rendering the immersive media data corresponding to the current viewing posture, wherein when the current viewing posture is within the three-dimensional viewing space, the obtained immersive media data is rendered; or, when the current viewing posture has moved or is moving outside the three-dimensional viewing space, rendering the immersive media data according to processing information of the three-dimensional viewing space.

[0008] In an exemplary embodiment, obtaining immersive media data corresponding to the user's current viewing posture includes: determining a relationship between the current viewing posture and the three-dimensional viewing space, wherein the three-dimensional viewing space is a range allowing the user to move in the immersive media scene; and determining the immersive media data based on the relationship.

[0009] In an exemplary embodiment, determining the relationship between the current viewing posture and the three-dimensional viewing space includes: determining a position of the current viewing posture in or outside the three-dimensional viewing space; and determining the immersive media data according to the position.

[0010] In an exemplary embodiment, before rendering the immersive media data corresponding to the current viewing posture, the method includes: determining a media track or data box describing processing information of the three-dimensional viewing space in a media file according to a data box type or a timed metadata track sample entry type.

[0011] In an exemplary embodiment, determining a media track or data box describing the three-dimensional viewing space in a media file according to a data box type or a timed metadata track sample entry type includes one of the following methods: identifying a first three-dimensional viewing space data box in a file header of the media file according to a first data box type; or, identifying a second three-dimensional viewing space grouping data box in a file header of the media file according to a second grouping type; or, identifying a third three-dimensional viewing space data box in one or more media tracks in the media file according to a third data box type; or, identifying a fourth three-dimensional viewing space sample group description data box in a media track in the media file according to a fourth sample grouping type; or, identifying a fifth three-dimensional viewing space data box in a media track in the media file according to a fifth track group type, wherein one or more media tracks with the same track group identifier belong to the same three-dimensional viewing space; or, identifying a three-dimensional viewing space timed metadata track in the media file according to a sixth sample entry type, wherein the three-dimensional viewing space timed metadata track indicates the dynamically changing three-dimensional viewing space of the immersive media data.

[0012] In an exemplary embodiment, the information of the three-dimensional viewing space includes at least one of the following: the coordinate position of the three-dimensional viewing space in the spatial scene of the immersive media, the orientation of the three-dimensional viewing space in the spatial scene of the immersive media, the geometric structure of the three-dimensional viewing space, and the viewing direction in the three-dimensional viewing space, wherein the structure of the three-dimensional viewing space is constructed by combining one or more complex geometric structures, and each of the complex geometric structures is constructed by combining one or more basic structures; each of the basic structures corresponds to zero or one acquisition device or acquired view of the immersive media data.

[0013] In an exemplary embodiment, before rendering the immersive media data corresponding to the current viewing posture, the method includes: determining a media track or data box describing processing information of the three-dimensional viewing space in a media file according to a data box type or a timed metadata track sample entry type.

[0014] In an exemplary embodiment, the determining of the media track or data box describing the processing information of the three-dimensional viewing space in the media file according to the data box type or the timed metadata track sample entry type includes: identifying the three-dimensional viewing space processing data box through the seventh data box type, wherein the three-dimensional viewing space processing data box is included in the data box describing the three-dimensional viewing space, or is in the same upper-level data box as the data box describing the three-dimensional viewing space; or, the three-dimensional viewing space processing data box is included in the three-dimensional viewing space timed metadata track; or, identifying the three-dimensional viewing space processing timed metadata track in the media file according to the eighth sample entry type, the three-dimensional viewing space processing timed metadata track indicating the manner in which the three-dimensional viewing space dynamically changes is processed.

[0015] In an exemplary embodiment, the processing information of the three-dimensional viewing space includes at least one of the following: the number of options for the three-dimensional viewing space processing method, the device type for three-dimensional viewing space processing, the application type for three-dimensional viewing space processing, the three-dimensional viewing space processing method, and the identification of the three-dimensional viewing space.

[0016] In an exemplary embodiment, obtaining immersive media data corresponding to a user's current viewing posture includes: directly obtaining the immersive media data when it is determined that the user's current viewing posture is within the three-dimensional viewing space; or, obtaining the immersive media data corresponding to the current viewing posture based on processing information of the three-dimensional viewing space when it is determined that the user's current viewing posture is moving or has moved outside the three-dimensional viewing space.

[0017] In an exemplary embodiment, obtaining the immersive media data includes: determining a media presentation description file, wherein the media presentation description file includes a three-dimensional viewing space descriptor and / or a three-dimensional viewing space processing descriptor indicating viewing immersive media; and requesting to obtain the immersive media data corresponding to the current viewing posture based on the three-dimensional viewing space descriptor and / or the three-dimensional viewing space processing descriptor corresponding to the current viewing posture of the user.

[0018] In an exemplary embodiment, the three-dimensional viewing space descriptor of the immersive media includes at least one of the following: basic geometric structure, rotation direction of the basic geometric body, view or acquisition device identifier or index corresponding to the basic geometric body, rotation of the basic geometric body, viewing direction in the basic geometric body, basic geometric body combination method, complex combination method, and three-dimensional viewing space identifier.

[0019] In an exemplary embodiment, the immersive media 3D viewing space processing descriptor includes at least one of the following: a 3D viewing space processing device type, a 3D viewing space processing application type, a 3D viewing space processing method, and a 3D viewing space identifier.

[0020] According to another embodiment of the present invention, a device for processing immersive media data is provided, including: an acquisition unit for acquiring immersive media data corresponding to a user's current viewing posture, wherein a three-dimensional viewing space is used to limit the rendering of the user's viewing content; a rendering unit for rendering the immersive media data corresponding to the current viewing posture, wherein when the current viewing posture is within the three-dimensional viewing space, the acquired immersive media data is rendered; or, when the current viewing posture has moved or is moving outside the three-dimensional viewing space, the immersive media data is rendered according to processing information of the three-dimensional viewing space.

[0021] In an exemplary embodiment, the acquisition unit includes: a first determination module for determining the relationship between the current viewing posture and the three-dimensional viewing space, wherein the three-dimensional viewing space is the range allowing the user to move in the immersive media scene; and a second determination module for determining the immersive media data based on the relationship.

[0022] In an exemplary embodiment, the first determination module includes: a first determination submodule for determining a position of the current viewing posture in or outside the three-dimensional viewing space; and a second determination submodule for determining the immersive media data according to the position.

[0023] In an exemplary embodiment, the device also includes: a first determination unit, which is used to determine the media track or data box that describes the processing information of the three-dimensional viewing space in the media file according to the data box type or the timed metadata sample entry type before rendering the immersive media data corresponding to the current viewing posture.

[0024] In an exemplary embodiment, the first determination unit is further used to perform one of the following: identifying a first three-dimensional viewing space data box in a file header of the media file according to a first data box type; or, identifying a second three-dimensional viewing space grouping data box in a file header of the media file according to a second grouping type; or, identifying a third three-dimensional viewing space data box in one or more media tracks in the media file according to a third data box type; or, identifying a fourth three-dimensional viewing space sample group description data box in a media track in the media file according to a fourth sample grouping type; or, identifying a fifth three-dimensional viewing space data box in a media track in the media file according to a fifth track group type, wherein one or more media tracks with the same track group identifier belong to the same three-dimensional viewing space; or, identifying a three-dimensional viewing space timing metadata track in the media file according to a sixth sample entry type, wherein the three-dimensional viewing space timing metadata track indicates a dynamically changing three-dimensional viewing space of the immersive media data.

[0025] In an exemplary embodiment, the information of the three-dimensional viewing space includes at least one of the following: the coordinate position of the three-dimensional viewing space in the spatial scene of the immersive media, the orientation of the three-dimensional viewing space in the spatial scene of the immersive media, the geometric structure of the three-dimensional viewing space, and the viewing direction in the three-dimensional viewing space, wherein the structure of the three-dimensional viewing space is constructed by combining one or more complex geometric structures, and each of the complex geometric structures is constructed by combining one or more basic structures; each of the basic structures corresponds to zero or one acquisition device or acquired view of the immersive media data.

[0026] In an exemplary embodiment, the device also includes: a second determination unit for determining a media track or data box describing processing information of the three-dimensional viewing space in a media file according to a data box type or a timed metadata track sample entry type before rendering the immersive media data corresponding to the current viewing posture.

[0027] In an exemplary embodiment, the second determination unit includes: an identification module for identifying the three-dimensional viewing space processing data box through the seventh data box type, wherein the three-dimensional viewing space processing data box is included in the data box describing the three-dimensional viewing space, or is in the same upper-level data box as the data box describing the three-dimensional viewing space; or, the three-dimensional viewing space processing data box is included in the three-dimensional viewing space timing metadata track; or, according to the eighth sample entry type, the three-dimensional viewing space processing timing metadata track in the media file is identified, and the three-dimensional viewing space processing timing metadata track indicates the manner of three-dimensional viewing space processing of the dynamic changes of the three-dimensional viewing space.

[0028] In an exemplary embodiment, the processing information of the three-dimensional viewing space includes at least one of the following: the number of options for the three-dimensional viewing space processing method, the device type for three-dimensional viewing space processing, the application type for three-dimensional viewing space processing, the three-dimensional viewing space processing method, and the identification of the three-dimensional viewing space.

[0029] In an exemplary embodiment, the acquisition unit includes: a first acquisition module for directly acquiring the immersive media data when it is determined that the user's current viewing posture is within the three-dimensional viewing space; or a second acquisition module for acquiring the immersive media data corresponding to the current viewing posture based on processing information of the three-dimensional viewing space when it is determined that the user's current viewing posture is moving or has moved outside the three-dimensional viewing space.

[0030] In an exemplary embodiment, the first acquisition module or the second acquisition module is further used to: determine a media presentation description file, wherein the media presentation description file includes a three-dimensional viewing space descriptor and / or a three-dimensional viewing space processing descriptor indicating viewing immersive media; and request to obtain the immersive media data corresponding to the current viewing posture according to the three-dimensional viewing space descriptor and / or the three-dimensional viewing space processing descriptor corresponding to the user's current viewing posture.

[0031] In an exemplary embodiment, the three-dimensional viewing space descriptor of the immersive media includes at least one of the following: basic geometric structure, rotation direction of the basic geometric body, view or acquisition device identifier or index corresponding to the basic geometric body, rotation of the basic geometric body, viewing direction in the basic geometric body, basic geometric body combination method, complex combination method, and three-dimensional viewing space identifier.

[0032] In an exemplary embodiment, the immersive media 3D viewing space processing descriptor includes at least one of the following: a 3D viewing space processing device type, a 3D viewing space processing application type, a 3D viewing space processing method, and a 3D viewing space identifier.

[0033] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0034] According to another embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0035] Through the embodiments of the present invention, by obtaining immersive media data corresponding to the user's current viewing posture, wherein a three-dimensional viewing space is used to limit the rendering of the user's viewing content; rendering the immersive media data corresponding to the current viewing posture, wherein when the current viewing posture is within the three-dimensional viewing space, the obtained immersive media data is rendered; or when the current viewing posture has moved or is moving outside the three-dimensional viewing space, the method of rendering the immersive media data according to the processing information of the three-dimensional viewing space, since when the user moves outside the three-dimensional viewing space, the immersive media data can be rendered according to the processing information of the three-dimensional viewing space, even if the user moves outside the three-dimensional viewing space, high-quality visual content can be provided to the user quickly and efficiently. Therefore, it is possible to solve the problems affecting the rendering and presentation of the visual content viewed by the user, and achieve fast, efficient and high-quality reconstruction and rendering of the visual content during the viewing process to meet the optimal immersive experience effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic structural diagram of an execution device of a method for processing immersive media data according to an embodiment of the present invention;

[0037] Figure 2 is a flowchart of a method for processing immersive media data according to an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of a three-dimensional viewing space according to a method for processing immersive media data according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of the system structure of a method for processing immersive media data according to an embodiment of the present invention;

[0040] Figure 5 is a flowchart of a method for processing immersive media data according to an embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of an iOS file according to a method for processing immersive media data according to an embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of an iOS file according to a method for processing immersive media data according to an embodiment of the present invention;

[0043] Figure 8 is a schematic diagram of an iOS file according to a method for processing immersive media data according to an embodiment of the present invention;

[0044] Figure 9 is a schematic diagram of an iOS file according to a method for processing immersive media data according to an embodiment of the present invention;

[0045] Figure 10 is a schematic diagram of an iOS file according to a method for processing immersive media data according to an embodiment of the present invention;

[0046] Figure 11 is a flowchart of a method for processing immersive media data according to an embodiment of the present invention;

[0047] Figure 12 FIG. 4 is a schematic structural diagram of an immersive media data processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with embodiments.

[0049] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0050] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG is a hardware structure block diagram of a mobile terminal of an immersive media data processing method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data are shown. The mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0051] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for processing immersive media data in the embodiment of the present invention. The processor 102 executes the computer program stored in the memory 104 to execute various functional applications and data processing, that is, to implement the above-mentioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the mobile terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] In this embodiment, a method for processing immersive media data is provided. Figure 2 FIG. 1 is a flowchart of a method for processing immersive media data according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0054] Step S202, obtaining immersive media data corresponding to the user's current viewing posture, wherein a three-dimensional viewing space is used to limit the rendering of the user's viewing content;

[0055] Step S204: Rendering the immersive media data corresponding to the current viewing posture, wherein the acquired immersive media data is rendered when the current viewing posture is within the three-dimensional viewing space; or, when the current viewing posture has moved or is moving outside the three-dimensional viewing space, rendering the immersive media data based on processing information of the three-dimensional viewing space.

[0056] It should be noted that the three-dimensional viewing space in this application is a space that allows users to move. In layman's terms, a virtual camera is set up in the three-dimensional viewing space, and the virtual camera can capture the content in the three-dimensional viewing space and feed it back to the user in the real space. If the user moves or rotates in the real space, the virtual camera moves or rotates in the three-dimensional viewing space. The three-dimensional viewing space is the range within which the system recommends that the user, or the virtual camera, move. The user or the virtual camera can also move outside the three-dimensional viewing space, and it is not necessary to move within the three-dimensional viewing space. The above-mentioned use of the three-dimensional viewing space to limit the rendering of the user's viewing content means that after the virtual camera captures the content in the three-dimensional viewing space, it will be passed to the user for viewing. At this time, the content captured by the virtual camera needs to be rendered. In other words, the content viewed by the user is rendered.

[0057] The obtaining of the immersive media data corresponding to the user's current viewing posture includes: determining a relationship between the current viewing posture and the three-dimensional viewing space, wherein the three-dimensional viewing space is a virtual space range allowing the user to move; and determining the immersive media data based on the relationship.

[0058] Determining the relationship between the current viewing posture and the three-dimensional viewing space includes: determining a position of the current viewing posture in or outside the three-dimensional viewing space; and determining the three-dimensional viewing space of the immersive media data according to the position.

[0059] Before rendering the immersive media data corresponding to the current viewing posture, the method includes: determining a media track or a data box describing the processing information of the three-dimensional viewing space in a media file according to a data box type or a timed metadata sample entry type.

[0060] Among them, determining the media track or data box describing the three-dimensional viewing space in the media file according to the data box type or the timed metadata track sample entry type includes one of the following methods: identifying the first three-dimensional viewing space data box in the file header of the media file according to the first data box type; or, identifying the second three-dimensional viewing space grouping data box in the file header of the media file according to the second grouping type; or, identifying the third three-dimensional viewing space data box in one or more media tracks in the media file according to the third data box type; or, identifying the fourth three-dimensional viewing space sample group description data box in a media track in the media file according to the fourth sample grouping type; or, identifying the fifth three-dimensional viewing space data box in a media track in the media file according to the fifth track group type, wherein one or more media tracks with the same track group identifier belong to the same three-dimensional viewing space; or, identifying the three-dimensional viewing space timed metadata track in the media file according to the sixth sample entry type, and the three-dimensional viewing space timed metadata track indicates the dynamically changing three-dimensional viewing space of the immersive media data.

[0061] In which, the processing information of the three-dimensional viewing space includes at least one of the following: the coordinate position of the three-dimensional viewing space in the spatial scene of the immersive media, the orientation of the three-dimensional viewing space in the spatial scene of the immersive media, the geometric structure of the three-dimensional viewing space, and the viewing direction in the three-dimensional viewing space, wherein the structure of the three-dimensional viewing space is constructed by combining one or more complex geometric structures, and each of the complex geometric structures is constructed by combining one or more basic structures; each of the basic structures corresponds to zero or one acquisition device or acquired view of the immersive media data.

[0062] Before rendering the immersive media data corresponding to the current viewing posture, the method includes: determining a media track or data box describing the processing information of the three-dimensional viewing space in a media file according to a data box type or a timed metadata track sample entry type.

[0063] Among them, the determining of the media track or data box describing the processing information of the three-dimensional viewing space in the media file according to the data box type or the timed metadata track sample entry type includes: identifying the three-dimensional viewing space processing data box through the seventh data box type, wherein the three-dimensional viewing space processing data box is included in the data box describing the three-dimensional viewing space, or is in the same upper-level data box as the data box describing the three-dimensional viewing space; or, the three-dimensional viewing space processing data box is included in the three-dimensional viewing space timed metadata track; or, identifying the three-dimensional viewing space processing timed metadata track in the media file according to the eighth sample entry type, the three-dimensional viewing space processing timed metadata track indicating the manner in which the three-dimensional viewing space dynamically changes is processed.

[0064] The processing information of the three-dimensional viewing space includes at least one of the following: the number of options for the three-dimensional viewing space processing method, the device type for three-dimensional viewing space processing, the application type for three-dimensional viewing space processing, the three-dimensional viewing space processing method, and the identification of the three-dimensional viewing space.

[0065] Among them, obtaining the immersive media data corresponding to the user's current viewing posture includes: when it is determined that the user's current viewing posture is within the three-dimensional viewing space, directly obtaining the immersive media data; or, when it is determined that the user's current viewing posture is moving or has moved outside the three-dimensional viewing space, obtaining the immersive media data corresponding to the current viewing posture based on processing information of the three-dimensional viewing space.

[0066] Wherein, obtaining the immersive media data includes: determining a media presentation description file, wherein the media presentation description file includes a three-dimensional viewing space descriptor and / or a three-dimensional viewing space processing descriptor indicating viewing immersive media; and requesting to obtain the immersive media data corresponding to the current viewing posture according to the three-dimensional viewing space descriptor and / or the three-dimensional viewing space processing descriptor corresponding to the current viewing posture of the user.

[0067] Among them, the three-dimensional viewing space descriptor of the immersive media includes at least one of the following: basic geometric body structure, rotation direction of the basic geometric body, view or acquisition device identifier or index corresponding to the basic geometric body, rotation of the basic geometric body, viewing direction in the basic geometric body, basic geometric body combination method, complex combination method, and three-dimensional viewing space identifier.

[0068] The immersive media's 3D viewing space processing descriptor includes at least one of the following: the device type of the 3D viewing space processing, the application type of the 3D viewing space processing, the 3D viewing space processing method, and the identification of the 3D viewing space. These steps address issues affecting the rendering and presentation of visual content for users, achieving fast, efficient, and high-quality reconstruction and rendering of visual content during viewing, ensuring an optimal immersive experience.

[0069] This application can be applied to an immersive experience. A user views a scene in a three-dimensional viewing space by wearing a display device. The user can move in real space, which is mapped to the movement of the user's virtual character in the three-dimensional viewing space. The immersive media data captured by the corresponding virtual character is played on the display device. During this process, if the user's current viewing posture in the three-dimensional viewing space exceeds the three-dimensional viewing space, the immersive media data is rendered based on the processing information of the three-dimensional viewing space.

[0070] Among them, the three-dimensional viewing space refers to the range within which the user's virtual character can move in the immersive media scene. The three-dimensional viewing posture refers to the position and viewing direction (virtual camera direction) of the user's corresponding virtual character in the three-dimensional viewing space. If the virtual character exceeds the range of the three-dimensional viewing space, problems such as a black screen may be caused in related technologies. In this application, if the user's virtual character exceeds the range of the three-dimensional viewing space, the immersive media data can be rendered according to the processing information of the three-dimensional viewing space, thereby achieving fast, efficient, and high-quality reconstruction and rendering of visual content during the viewing process. Solve the problem of black screen.

[0071] The following describes how to construct a three-dimensional viewing space and how to obtain immersive media data with reference to specific examples.

[0072] Figure 3 It is a schematic diagram of an optional three-dimensional viewing space structure position. In the example scenario, immersive video acquisition devices (cameras) are placed in different positions to capture video data in the scene. Based on the position of the acquisition device, one or more three-dimensional viewing spaces can be formed. The three-dimensional viewing space can be a basic geometric body represented by a separate cube, sphere, cylinder, etc., or it can be a complex geometric body composed of multiple basic geometric bodies. When the user moves in the three-dimensional viewing space, the display device can obtain the media data corresponding to the user's window, reconstruct and render the image in the window, and quickly switch the content presentation in the user's window; when the user moves out of the three-dimensional viewing space, because there is no media data collected by the corresponding device, the display device will not be able to display the corresponding visual content in the user's window, or can only display low-quality visual content. Figure 3 The three-dimensional viewing spaces 1-3 in FIG are different situations.

[0073] At the same time, in a three-dimensional viewing space, the user's viewing direction may also be restricted by the position and orientation of the acquisition device.

[0074] Figure 4 The present invention is a structural diagram of an optional immersive media data processing system.

[0075] like Figure 4 As shown, the system includes a transmission server 402 and a user terminal 404.

[0076] The transmission server 402 at least includes a storage module 402-1 and a transmission module 402-2.

[0077] The storage module 402 - 1 is used to store the media files produced in the production server 10 .

[0078] Transmission module 402-2 is used to receive a request message from a user terminal or send a stored media file. The above-mentioned receiving or sending can be achieved through a wireless network provided by a communication provider, a locally established wireless local area network, or a wired method;

[0079] The user terminal 404 at least includes a transmission module 404 - 1 , a decoding and decapsulation module 404 - 2 , a media processing module 404 - 3 and a display module 404 - 4 .

[0080] Transmission module 404-1 is used to receive media files sent by media server 402, or send a request message to server 402, such as requesting media file download;

[0081] The decoding and decapsulation module 404-2 is used to decode the media file received by the decapsulation and transmission module;

[0082] The media processing module 404-3 reconstructs and renders the media data output by the decoding and decapsulation module 404-2 based on the user's current viewing position, viewing direction, and other information;

[0083] The display module 404 - 4 is used to present the visual content of the user's current window to the user.

[0084] This embodiment provides a processing / playback process of a media system based on a three-dimensional viewing space, and a representation of a file format, such as Figure 5 As shown, the process includes the following steps:

[0085] Step S501: When a user's viewing posture (viewing position, viewing direction) changes, the user terminal obtains a corresponding media file based on the three-dimensional viewing space range to which the user's current posture information belongs. The media file includes immersive media data.

[0086] In step S502, the user terminal decapsulates and decodes the acquired media file, reconstructs the visual content in the current window according to the user's current viewing position, viewing direction and other information, and renders the content to present to the user.

[0087] It should be noted that, in an embodiment of the present invention, one implementation method is to store omnidirectional video data in a file based on the ISO (International Organization for Standardization) basic media file format. The ISO basic media file formats such as the restricted scheme information box, track reference box, and track group box can be operated with reference to the MPEG-4 Part 12 ISO Base Media File Format developed by the ISO / IEC JTC1 / SC29 / WG11 Moving Picture Experts Group (MPEG). The projection and encapsulation steps of the omnidirectional video and its basic format can be operated with reference to the MPEG-1 Part 2 OMAF (Omnidirectional Media Format) developed by the ISO / IEC JTC1 / SC29 / WG11 Moving Picture Experts Group (MPEG).

[0088] In addition, all data in the ISO base file format is contained within boxes. For example, the ISO base file format, represented by the MP4 file, consists of several boxes, each with a type and length, and can be considered a data object. A box can contain another box, called a container box. An MP4 file first has one and only one "ftyp" box, which identifies the file format and contains information about the file. This is followed by one and only one "MOOV" box (Movie Box), a container box whose child boxes contain media metadata. The media data in an MP4 file is contained within an "mdat" box (Media Data Box), also a container box. This box can contain multiple or none (when all media data references other files). The structure of the media data is described by metadata. To further support metadata descriptions of the media, a "meta" box, also a container box, is optionally used to describe general or additional non-scheduled metadata. A media can be composed of one or more tracks. Each track is a media sequence that changes over time. A track contains a continuous set of samples.

[0089] Furthermore, the timed metadata track is a mechanism in the ISO Base Media File Format (ISOBMFF) to establish timed metadata associated with a specific sample. Timed metadata is less coupled with the media data and is usually "descriptive".

[0090] Specifically, data boxes or sample entries are defined in the media file to describe the scene's 3D viewing space. 3D viewing spaces that are constant, infrequently changing, or frequently changing can be contained in different layers, media tracks, or containers. 3D viewing spaces can be described at the file or track level. Furthermore, a scene can contain one or more independent geometric objects representing the 3D viewing space.

[0091] Optionally, the specific description of the three-dimensional viewing space may be one of the following:

[0092] Method 1: Describe the three-dimensional viewing space at the file level, and define the three-dimensional viewing space data box (ViewingSpaceBox) to be included in the file-level Metabox ( Figure 6 The following describes the optional implementation of the 3D viewing space data box.

[0093] ViewingSpaceBox (3D viewing space data box)

[0094] Box Type:'vwsp'

[0095] Container:MetaBox

[0096] Mandatory:No

[0097] Quantity:Zero or one

[0098] grammar

[0099] aligned(8)class ViewingSpaceBox extends FullBox('vwsp',0,flags){signed int(8)num_viewing_space; / / Optional. If there is only one viewing space in a media track or media track group, this may not be applicable.

[0100]

[0101] Semantics:

[0102] num_viewing_space, indicating the number of 3D viewing spaces corresponding to the media file;

[0103] viewing_space_id, indicating the 3D viewing space ID;

[0104] Specifically, the structure of the three-dimensional viewing space is represented by ViewingSpaceStruct(), and the spatial structure is constructed by combining one or more complex geometric bodies, usually using the CSG (Constructive Solid Geometry) method; the complex geometric body is obtained by orderly adding and combining basic geometric bodies (cubes, cylinders, etc.), usually using the CSG method or sequential interpolation method.

[0105] Specifically, the basic geometric bodies are common shapes such as cubes, spheres and hemispheres, which are described below in conjunction with optional implementations of basic geometric body structures. Cuboidstruct(), SpheroidStruct() and HalfspaceStruct() represent cubes, spheres and hemispheres respectively.

[0106]

[0107]

[0108] Semantics

[0109] center_x, center_y, and center_z respectively indicate the position of the center point of the geometric structure in the coordinate system;

[0110] size_x, size_y, size_z, respectively indicate the side lengths of the cube in the x, y, and z directions;

[0111] radius_x, radius_y, radius_z, respectively indicate the radius of the sphere in the x, y, and z dimensions;

[0112] normal_x, normal_y, normal_z, respectively, indicate the plane normals that define the hemisphere;

[0113] camera_inferred_flag, specified by SpaceShapeStruct(), indicates whether the position of the simple geometric body corresponds to the acquisition device; 0 means it is independent of the acquisition device position and the center point position needs to be defined by yourself; 1 means it is related to the acquisition device and the acquisition device position information can be used;

[0114] distance, indicating the distance from the origin along the normal vector to the plane of the hemisphere.

[0115] Specifically, the complex geometric body is described by a shape space structure (ShapeSpaceStruct), which is constructed from one or more basic geometric bodies. The following describes the optional implementation of the three-dimensional viewing space structure.

[0116] grammar

[0117]

[0118]

[0119] Semantics

[0120] num_primitive_shape, indicating the number of primitive shapes that make up the three-dimensional viewing space;

[0121] primitive_shape_operation, indicating the operation mode of the basic geometric shapes that make up the three-dimensional viewing space; when it is 0, it indicates that the basic geometric shapes are added together using the CSG mode to form a complex geometric shape; when it is 1, it indicates that the basic geometric shapes are interpolated along the path formed by the center of the basic geometric shapes to form a complex geometric shape;

[0122] camera_inferred_flag, when 1, indicates that the position and orientation of the basic geometry are corresponding to the acquisition device, where the acquisition device corresponds to the viewpoint index number; when 0, indicates that the position and orientation of the basic geometry do not correspond to the acquisition device;

[0123] viewing_space_shape_type indicates the basic geometric shape of the 3D viewing space. The specific shape types are described in the following table;

[0124]

[0125]

[0126] distance_scale, which indicates the scale of the bounding box distance size of the underlying geometry;

[0127] view_id indicates the ID of the viewpoint of the camera corresponding to the basic geometry. Through this ID, the media track where the media data corresponding to the viewpoint is located can be located.

[0128] Specifically, the three-dimensional viewing space structure is described by a three-dimensional viewing space structure (ViewingSpaceStruct), which is constructed by combining one or more basic complex geometric bodies. The following describes the three-dimensional viewing space structure in conjunction with an optional implementation.

[0129] grammar

[0130]

[0131] num_shape_space, indicating the number of complex geometric bodies required to form a three-dimensional viewing space structure;

[0132] operation_type indicates the CSG operation mode for combining the geometry into a three-dimensional viewing space, as shown in the following table:

[0133]

[0134] Method 2: Describe the three-dimensional viewing space at the file level. There may be multiple media spaces in a media file. The media data of one or more media tracks (such as multi-viewpoint data, point cloud data) corresponds to a three-dimensional viewing space, that is, the user can view the visual content rendered by the corresponding media data in the three-dimensional viewing space. Group the media tracks according to the three-dimensional viewing space. All media tracks in the same group belong to the same three-dimensional viewing space. Use the entity grouping method in ISO BMFF to describe the three-dimensional viewing space corresponding to the media track group ( Figure 7 shown).

[0135] Specifically, the three-dimensional viewing space is described by extending EntityToGroupBox, which is included in GroupsListBox under the file-level Metabox, wherein grouping_type is 'vspg'; this will be explained below in conjunction with the optional implementation of ViewingSpaceGroupBox.

[0136]

[0137] Method 3: Describe the three-dimensional viewing space at the track level. A media track may correspond to one or more three-dimensional viewing spaces. Figure 8 The three-dimensional viewing space is described in detail below with reference to an optional embodiment.

[0138] ViewingSpaceBox (3D viewing space data box)

[0139] Box Type:'vwsp'

[0140] Container:Sample Entry

[0141] Mandatory:No

[0142] Quantity:Zero or one

[0143] grammar

[0144]

[0145]

[0146] Specifically, the media track grouping may be grouping the media tracks where the atlas data is located, and describing information about the three-dimensional viewing space corresponding to the media tracks in the sample entry of the media tracks where the atlas data is located.

[0147] Optionally, based on a specific application or specific media data, the sampleentry in the corresponding media track is extended. For example, the media file entry based on volumetric video data is VolumetricVisualSampleEntry, which is described as an SEI message in the configuration data box (VPCCConfigurationBox).

[0148] Method 4: Describing the 3D viewing space at the track level. A media track may correspond to one or more 3D viewing spaces, or the structure of the 3D viewing space may change at low frequencies during playback. Sample grouping is used to describe the 3D viewing space that each sample may correspond to. One sample group corresponds to one 3D viewing space, and grouping_type is 'vssg'. This is described below with reference to an optional embodiment.

[0149]

[0150] Optionally, based on specific applications or specific media data, the samplegrouping in the corresponding media track is extended. For example, based on the sample grouping of type 'vaps' for volumetric video data, the 3D viewing space is described as an SEI message in the sample grouping entry (V3CAtlasParamSampleGroupDescriptionEntry).

[0151] Method 5: Describe the 3D viewing space at the track level. A media file may contain multiple media spaces. The media data (such as multi-viewpoint data, point cloud data) of one or more media tracks corresponds to a 3D viewing space. That is, the user can view the visual content rendered by the corresponding media data in the 3D viewing space. The media tracks are grouped by defining a track group with track_group_type as 'vptg'. Figure 9As shown), all media tracks in the same group (containing the same track_gourp_id) belong to the same 3D viewing space. The type of the list box group GroupListBox is 'vspg'

[0152] Specifically, each group contains exactly one base track. For example, the media track containing the basic view in multi-view media data is the base track. The sample entry for the base track describes the 3D viewing space corresponding to the media track group. This is described below with reference to an optional embodiment.

[0153] ViewingSpaceBox (3D viewing space data box)

[0154] Box Type:'vwsp'

[0155] Container:Sample Entry

[0156] Mandatory:No

[0157] Quantity:Zero or one

[0158] grammar

[0159]

[0160] Method 6: Define a media track in the media file to describe or store various parameter information of the media. Use the sample to carry the information description of the NAL unit, and store the SEI message describing the three-dimensional viewing space at the bitstream layer in the corresponding sample.

[0161] As time goes by, the 3D viewing space will change dynamically due to changes in the position of the acquisition device or the 3D viewing space recommended by the director. This embodiment provides a method for representing 3D viewing space information based on variable information, such as Figure 10 shown.

[0162] Specifically, by specifying the 3D viewing space in the timed metadata track, the track sample entry of the 3D viewing space timed metadata track is identified by the 'vrsp' type, which is described below in conjunction with an optional implementation of the 3D viewing space structure.

[0163] grammar

[0164]

[0165] Optionally, each sample corresponds to one or more 3D viewing spaces, and the 3D viewing space information of each sample is provided by VRSpaceSample();

[0166]

[0167] static_vr_space_flag, when it is 1, indicates that the 3D viewing space is defined in the sample of this sample entry; when it is 0, it indicates that the 3D viewing space of all subsequent samples remains unchanged and references this sample entry.

[0168] num_viewing_space, indicating the number of 3D viewing spaces;

[0169] viewing_space_id, indicating the identifier of the 3D viewing space;

[0170] Optionally, if one timed metadata track can only describe the dynamic changes of one 3D viewing space, when the media file corresponds to multiple 3D viewing spaces, multiple timed metadata tracks may exist to describe the changes of the multiple 3D viewing spaces.

[0171] Optionally, the dynamic change of the three-dimensional viewing space is related to the dynamic change of the acquisition device, and the sample entry (CameraInfoSampleEntry) and the sample in the timing metadata track describing the camera dynamic information can be expanded.

[0172] In addition, the dynamic viewpoint timed metadata track can reference a track or track group by referencing a track reference box (Track Reference Box) of type 'cdsc'.

[0173] Due to the orientation of the media data acquisition device in space, there may be viewing issues. The viewing direction or viewing range in the three-dimensional viewing space is described by extending ViewingSpaceStruct.

[0174] Due to the position and orientation of the media data acquisition device in space, the basic geometric shape of the three-dimensional viewing space corresponding to the acquisition device may rotate in the coordinate system, such as the edges of a cube are not parallel to the coordinate axes.

[0175] Specifically, how the basic geometric body rotates in the coordinate system is described by a rotation structure (ShapeRotationStruct), which is explained below in conjunction with an optional implementation of a three-dimensional viewing space structure.

[0176]

[0177] Semantics

[0178] shape_rotation_x, shape_rotation_y, shape_rotation_z, respectively, indicate the x, y, z components of the rotation quaternion used for the underlying geometry.

[0179] Optionally, the rotation direction of the basic geometric body can also be indicated by Euler angle rotation, that is, shape_rotation_x, shape_rotation_y, shape_rotation_z, indicating the angles of rotation along the x, y, and z coordinate axes respectively.

[0180] Due to the position and orientation of the media data acquisition device in space, objects in the scene may not be captured from all angles. When the user moves freely while watching the video, the user's movable position range and rotatable viewing direction may be limited. That is, when the user's movement causes the visual content in the current window to be reconstructed, the viewing direction and viewing range exceed the limit, and the visual content in the window cannot be effectively reconstructed and rendered, resulting in content fading and a lack of immersion.

[0181] Specifically, the viewing direction or viewing range structure is described by a viewing direction constraint structure (ViewingDirectionConstrainStruct), which is described below in conjunction with an optional implementation of a three-dimensional viewing space structure based on the viewing direction and viewing range.

[0182]

[0183] Semantics

[0184] viewing_direction_center_x, viewing_direction_center_y, viewing_direction_center_z, respectively, indicate the quaternion components x, y, z of the center of the proposed viewing direction in the base geometry;

[0185] viewing_direction_yaw_range, viewing_direction_pitch_range, respectively indicate half the yaw range and pitch range of the suggested viewing direction in the base geometry;

[0186] guard_band_present_flag, which is 1 to indicate that the base geometry has a guard band, and 0 to indicate that the base geometry has no guard band;

[0187] guard_band_direction_size indicates the size of the guard band in the viewing direction within the underlying geometry, expressed in degrees.

[0188] Specifically, in consideration of viewing direction restrictions and geometric body rotation, SpaceShapeStruct() needs to be extended, which is explained below in conjunction with the optional implementation of a three-dimensional viewing space structure.

[0189]

[0190]

[0191] Specifically, the three-dimensional viewing space structure is described by a three-dimensional viewing space structure (ViewingSpaceStruct), which is constructed by combining one or more basic complex geometric bodies. The following describes the three-dimensional viewing space structure in conjunction with an optional implementation.

[0192] grammar

[0193]

[0194] Regarding the viewer's handling of the viewing process, moving into or out of the 3D viewing space, different operation types may be used based on different device types and application types. For example, the director recommends that the user render the scene when moving out of the 3D viewing space. This embodiment provides a method for representing the 3D viewing space rendering process and file format. The process includes the following steps:

[0195] Step S1: When the user's viewing posture (viewing position, viewing direction) changes, the user terminal obtains the corresponding media file based on the user's posture change trajectory, the 3D viewing space range, and the processing method corresponding to the 3D viewing space;

[0196] In step S2, the user terminal decodes and decapsulates the acquired media file, reconstructs the visual content in the current window based on the user's current viewing posture and the processing method corresponding to the three-dimensional viewing space, and renders the content to the user for viewing.

[0197] Specifically, when the user's viewing posture is within the three-dimensional viewing space, the default is to directly obtain the corresponding media file; when the user's viewing posture is moving outside the three-dimensional viewing space or is already outside the three-dimensional viewing space, the required media file is determined and obtained according to the corresponding three-dimensional viewing space processing method required according to different scenarios.

[0198] Specifically, the information related to the 3D viewing space processing method can be described at the file level or the track level. The 3D viewing space processing method is directly related to the scope of the 3D viewing space. Different 3D viewing spaces may support different device types, application types, or viewing processing methods.

[0199] Optionally, the three-dimensional viewing space processing method is described at the file level, and the following optional methods are available:

[0200] Method 1: The defined 3D viewing space processing data box (ViewingSpaceHandlingBox) is included in the file-level Metabox, and can be included in the same data box container as the 3D viewing space data box (ViewingSpaceBox).

[0201] Specifically, a ViewingSpaceHandlingBox (three-dimensional viewing space processing data box) of type 'vsph' is defined to describe a processing method based on the three-dimensional viewing space, which will be explained below in conjunction with optional implementations.

[0202]

[0203]

[0204] The specific syntax is as follows:

[0205] num_viewing_space, indicating the number of 3D viewing spaces in the scene;

[0206] viewing_space_id, indicating the identifier of the 3D viewing space to which this processing method applies;

[0207] num_handling_options: indicates the number of 3D viewing space processing options. When it is 0, it means that no 3D viewing space processing option is provided, and the target device can select an appropriate processing option based on the 3D viewing space information.

[0208] handling_device_class, indicating the device type for 3D viewing space handling (the specific values ​​are described in the table below) (all devices support 6DOF position tracking, and sometimes user movement is related to playback). In the same 3D viewing space, there should not be duplicate values ​​in a conformant bitstream. If the type value is 0, the value of i+1 should be num_handling_options.

[0209]

[0210] handling_application_class: indicates the application type of the 3D viewing space processing (the specific values ​​are described in the following table). In the same 3D viewing space, there should be no duplicate values ​​in a conformant bitstream. If the type value is 0, the value of i+1 should be num_handling_options;

[0211]

[0212]

[0213] handling_method: indicates the value of the 3D viewing space processing method (the specific values ​​are described in the following table). In the same 3D viewing space, there should be no repeated values ​​in a conformant bitstream. If the type value is 0, the value of i+1 should be num_handling_options;

[0214]

[0215] Method 2: A media file may contain multiple media spaces. Media tracks are grouped according to the 3D viewing space. All media tracks in the same group belong to the same 3D viewing space. The entity grouping method in ISO BMFF is used to describe the 3D viewing space processing method corresponding to the media track group.

[0216] Specifically, the 3D viewing space and its corresponding 3D viewing space processing method are described by extending EntityToGroupBox, which is included in GroupsListBox under the file-level Metabox, where grouping_type is 'vspg'; the following describes the optional implementation of ViewingSpaceGroupBox.

[0217]

[0218] Optionally, the three-dimensional viewing space processing method is described at the track level, and the following optional methods are available:

[0219] Method 1: A media track may correspond to one or more 3D viewing spaces, and the 3D viewing space handling information is described in the SampleEntry of the media track. That is, ViewingSpaceHandlingBox() is included in the SampleEntry, and its number is 0 or 1.

[0220] Method 2: Describing the 3D viewing space at the track level. A media track may correspond to one or more 3D viewing spaces, or the position and structure of a 3D viewing space may change at low frequencies during playback. Sample grouping is used to describe the 3D viewing space that each sample may correspond to and how it is processed. One sample group corresponds to one 3D viewing space, and grouping_type is 'vssg'. This is described below with reference to an optional embodiment.

[0221]

[0222]

[0223] Optionally, based on specific applications or specific media data, the samplegrouping in the corresponding media track is extended. For example, based on the sample grouping of type 'vaps' for volumetric video data, the 3D viewing space processing method is described as an SEI message in the sample group entry (V3CAtlasParamSampleGroupDescriptionEntry).

[0224] Method 3: A media file may contain multiple media spaces, and the media data of one or more media tracks (such as multi-viewpoint data and point cloud data) corresponds to a 3D viewing space. By defining a track group with track_group_type set to 'vptg', media tracks are grouped based on the 3D viewing space. All media tracks in the same group (containing the same track_group_id) belong to the same 3D viewing space.

[0225] Specifically, the media track grouping may be grouping the media tracks where the atlas data is located, and describing the media processing mode of the three-dimensional viewing space corresponding to the media track in the sample entry of the media track where the atlas data is located.

[0226] Specifically, if there is only one base track in each group, such as the media track containing the basic view in the multi-view media data is the base track, the sample entry of the base track describes the three-dimensional viewing space corresponding to the media track group.

[0227] Method 4: In the media track, there may be a media track that only stores media parameter information. By using the sample to carry the information description of the NAL unit, the description metadata of the 3D viewing space processing method can be stored in the corresponding sample as an SEI message.

[0228] When the 3D viewing space is constantly changing due to changes in the position of the acquisition device, the processing method of the 3D viewing space may also be constantly changing, or it may change due to the director's plot arrangement, etc., then metadata describing the changing 3D viewing space processing method is included in the timed metadata track.

[0229] Specifically, the 3D viewing space corresponds to the 3D viewing space processing mode, and the corresponding 3D viewing space processing mode is described in the 3D viewing space timed metadata track.

[0230] Specifically, VRSpaceSampleEntry() and its VRSpaceSample() are extended, as described below in conjunction with optional implementations. The timed metadata track for 3D viewing space processing is determined at the sample entry, and the extension of VRSpaceSampleEntry() can be one of the following methods:

[0231] grammar

[0232]

[0233] Method 2: Extend VRSpaceSampleEntry() to VRSpacehandlingEntry(), define 3D viewing space handling metadata based on the 3D viewing space defined in VRSpaceSampleEntry(), and use the 'vrsh' type to identify the sample entry of the 3D viewing space timing metadata track.

[0234] grammar:

[0235]

[0236] Optionally, each sample of the timed metadata media track describes one or more 3D viewing spaces, and metadata describing the 3D viewing space processing is included in each sample. The extension to VRSpaceSample() has one of the following methods:

[0237] Method 1: directly describe the metadata of the 3D viewing space processing corresponding to the 3D viewing space in VRSpaceSample().

[0238] grammar:

[0239]

[0240]

[0241] Method 2: Extend VRSpaceSample() to describe the metadata of the 3D viewing space processing corresponding to the 3D viewing space.

[0242] grammar:

[0243]

[0244] Semantics

[0245] space_handling_flag, when set to 1, indicates that the 3D viewing space processing is defined in the sample of this sample entry; when set to 0, indicates that the 3D viewing space processing method of all subsequent samples remains unchanged and refers to this sample entry;

[0246] viewing_space_idx, indicating the viewing space identification index;

[0247] handling_present: 0 indicates that the 3D viewing space follows the default processing method, and 1 indicates the processing method corresponding to the current 3D viewing space.

[0248] Optionally, if a timed metadata track can only describe the dynamic changes of a three-dimensional viewing space, when the media file corresponds to multiple three-dimensional viewing spaces, there may be multiple timed metadata tracks to describe the changes of multiple three-dimensional viewing spaces. In this case, one three-dimensional viewing space timed metadata corresponds to a processing method for the three-dimensional viewing space.

[0249] Alternatively, if the position and structure of the 3D viewing space change, but the corresponding viewing_space_id remains unchanged, and the intrinsic parameters of the acquisition device corresponding to the 3D viewing space remain unchanged, the processing of the 3D viewing space may remain unchanged. A separate timed metadata track can be used to describe the dynamic 3D viewing space processing. This optional implementation is explained below.

[0250] grammar

[0251]

[0252] Optionally, each sample corresponds to a 3D viewing space, and the information about the sample's 3D viewing space processing method is provided by VRSpaceHandlingSample();

[0253]

[0254]

[0255] This embodiment provides a method for a user terminal to obtain media data corresponding to the user's current posture ( Figure 11 The specific steps are as follows:

[0256] S1102: The user's viewing posture (viewing position, viewing direction) changes. The user terminal determines, based on the defined 3D viewing space, whether the user's latest viewing posture is within the 3D viewing space, is moving outside the 3D viewing space, or is already outside the 3D viewing space. The user terminal then determines whether corresponding media data exists locally based on the position of the user's viewing posture relative to the 3D viewing space and the corresponding processing method for the 3D viewing space.

[0257] S1104, determining whether there is media data corresponding to the user's current viewing posture locally;

[0258] S1106: If not, then search the MPD file for the media file location corresponding to the user's current posture, and request and download the corresponding media data from the server; if yes, then directly proceed to step S1108;

[0259] S1108: If yes, the user terminal reconstructs and renders the visual content within the user's current window based on the media data.

[0260] Specifically, in the Media Presentation Description (MPD) of Dynamic Adaptive Streaming over HTTP (DASH), the ViewingSpace element with the scheme identification attribute (@schemeIdUri) equal to "urn:mpeg:mpegI:miv:2020:vwsp" is called the three-dimensional viewing space (VWSP) descriptor, and the corresponding media data is requested and downloaded through the descriptor.

[0261] The three-dimensional viewing space is indicated at the MPD level or in the Adaptation Set using the VWSP descriptor. The VWSP descriptor cannot appear in both levels simultaneously; if it is expressed in a lower-level set, it is reused in the higher-level set. If the VWSP descriptor is absent, the viewing position may be either unavailable or freely movable in space.

[0262] As shown in Table 1 below, a table is provided for describing the semantics of the attributes of the 3D viewing space description sub-element.

[0263] Table 1

[0264]

[0265]

[0266] Optionally, different terminal devices and applications may process media files differently in the 3D viewing space. These different processing methods may require different media files to be selected for retrieval. For example, when a user's viewing posture changes (i.e., they leave the 3D viewing space), the corresponding media file for the next 3D viewing space may need to be retrieved. When retrieving media data corresponding to the 3D viewing space, the appropriate media file can be selected based on the 3D viewing space processing information in the MPD, and the corresponding visual content can be rendered based on the specific processing method.

[0267] Specifically, in the Media Presentation Description (MPD) of Dynamic Adaptive Streaming over HTTP (DASH), the ViewingSpaceHandling element with the scheme identification attribute (@schemeIdUri) equal to "urn:mpeg:mpegI:miv:2020:vwph" is called the three-dimensional viewing space processing mode (VWPH) descriptor, and the corresponding media data is requested and downloaded through the descriptor.

[0268] At the MPD level or in the Adaptation Set, the VWPH descriptor is used to indicate the processing method for the corresponding 3D viewing space. The VWPH descriptor cannot appear in two levels at the same time. If it is expressed in the lower level set, it is reused in the higher level.

[0269] As shown in Table 2 below, a table is provided for describing the semantics of the attributes of the 3D viewing space processing mode (VWPH) description sub-element.

[0270] Table 2

[0271]

[0272] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0273] This embodiment also provides an immersive media data processing device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0274] Figure 12 FIG. 1 is a structural block diagram of an immersive media data processing apparatus according to an embodiment of the present invention. Figure 12 As shown, the device includes

[0275] An acquisition unit 1202 is configured to acquire immersive media data corresponding to a user's current viewing posture, wherein a three-dimensional viewing space is used to restrict rendering of content viewed by the user;

[0276] The rendering unit 1204 is configured to render the immersive media data corresponding to the current viewing posture, wherein the acquired immersive media data is rendered when the current viewing posture is within the three-dimensional viewing space; or, when the current viewing posture has moved or is moving outside the three-dimensional viewing space, the immersive media data is rendered based on processing information of the three-dimensional viewing space.

[0277] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0278] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when running.

[0279] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0280] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0281] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0282] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0283] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0284] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for processing immersive media data, characterized in that: include: Obtaining immersive media data corresponding to the user's current viewing posture, wherein a three-dimensional viewing space is used to constrain the rendering of the user's viewing content, wherein the three-dimensional viewing space is a range within which the user is allowed to move in the immersive media scene; Before rendering the immersive media data corresponding to the current viewing posture, determining a 3D viewing space timed metadata track and a 3D viewing space processing timed metadata track describing processing information of the 3D viewing space in a media file, wherein the 3D viewing space timed metadata track indicates a dynamically changing 3D viewing space of the immersive media data, and the 3D viewing space processing timed metadata track indicates a manner of processing the dynamically changing 3D viewing space of the 3D viewing space; rendering immersive media data corresponding to the current viewing posture, wherein, when the current viewing posture has moved to or is moving to the three-dimensional viewing space, rendering the immersive media data according to the dynamically changing three-dimensional viewing space of the immersive media data indicated by the three-dimensional viewing space timing metadata track and the dynamically changing three-dimensional viewing space processing manner of the three-dimensional viewing space indicated by the three-dimensional viewing space processing timing metadata track; The acquiring of the immersive media data corresponding to the current viewing posture of the user includes: acquiring, based on processing information of the three-dimensional viewing space, the immersive media data corresponding to the current viewing posture when it is determined that the current viewing posture of the user is moving or has moved outside the three-dimensional viewing space; The step of obtaining the immersive media data according to the processing information of the three-dimensional viewing space includes: Determining a media presentation description file, wherein the media presentation description file includes a three-dimensional viewing space descriptor and / or a three-dimensional viewing space processing descriptor indicating viewing of immersive media; the three-dimensional viewing space descriptor is used to indicate the three-dimensional viewing space, and the three-dimensional viewing space processing descriptor is used to indicate a processing method for the three-dimensional viewing space; The immersive media data corresponding to the current viewing posture is acquired according to a three-dimensional viewing space descriptor and / or a three-dimensional viewing space processing descriptor corresponding to the current viewing posture of the user.

2. The method according to claim 1, characterized in that The obtaining of immersive media data corresponding to the user's current viewing posture includes: A relationship between the current viewing posture and the three-dimensional viewing space is determined.

3. The method according to claim 2, characterized in that Determining the relationship between the current viewing posture and the three-dimensional viewing space includes: Determining a position of the current viewing posture in the three-dimensional viewing space or outside the three-dimensional viewing space; The three-dimensional viewing space of the immersive media data is determined according to the position.

4. The method according to claim 1, wherein Before rendering the immersive media data corresponding to the current viewing posture, the method includes: A media track or data box describing the processing information of the three-dimensional viewing space in the media file is determined according to the data box type or the timed metadata track sample entry type.

5. The method according to claim 4, characterized in that Determining the timed metadata track or data box describing the three-dimensional viewing space in the media file according to the data box type or the timed metadata track sample entry type includes one of the following methods: identifying a first three-dimensional viewing space data box in a header of the media file according to a first data box type; Alternatively, identifying a second three-dimensional viewing space grouping data box in a file header of the media file according to a second grouping type; Alternatively, identifying a third three-dimensional viewing space data box in one or more media tracks in the media file according to a third data box type; Alternatively, identifying a fourth three-dimensional viewing space sample group description data box in a media track of the media file according to a fourth sample grouping type; Alternatively, identifying a fifth three-dimensional viewing space data box in a media track in the media file according to a fifth track group type, wherein one or more media tracks having the same track group identifier belong to the same three-dimensional viewing space; Alternatively, a three-dimensional viewing space timing metadata track in the media file is identified according to a sixth sample entry type.

6. The method according to any one of claims 1 to 5, characterized in that The processing information of the three-dimensional viewing space includes at least one of the following: The coordinate position of the three-dimensional viewing space in the spatial scene of the immersive media, the orientation of the three-dimensional viewing space in the spatial scene of the immersive media, the geometric structure of the three-dimensional viewing space, and the viewing direction in the three-dimensional viewing space, wherein the structure of the three-dimensional viewing space is constructed by combining one or more complex geometric structures, and each of the complex geometric structures is constructed by combining one or more basic structures; each of the basic structures corresponds to zero or one acquisition device or acquired view of the immersive media data.

7. The method according to claim 1, characterized in that Before rendering the immersive media data corresponding to the current viewing posture, the method includes: A timed metadata track or a data box describing processing information of the three-dimensional viewing space in a media file is determined according to a data box type or a timed metadata track sample entry type.

8. The method according to claim 7, characterized in that: The determining, according to the data box type or the timed metadata track sample entry type, a timed metadata track or a data box describing the processing information of the three-dimensional viewing space in the media file includes: Identifying the 3D viewing space processing data box by a seventh data box type, wherein the 3D viewing space processing data box is included in the data box describing the 3D viewing space, or is in the same parent data box as the data box describing the 3D viewing space; Alternatively, the 3D viewing space processing data box is included in the 3D viewing space timed metadata track; Alternatively, a three-dimensional viewing space processing timing metadata track in the media file is identified according to the eighth sample entry type.

9. The method according to claim 8, characterized in that The processing information of the three-dimensional viewing space includes at least one of the following: The number of options for 3D viewing space processing methods, the device type for 3D viewing space processing, the application type for 3D viewing space processing, the 3D viewing space processing method, and the identifier of the 3D viewing space.

10. The method according to claim 1, characterized in that: The three-dimensional viewing space descriptor of the immersive media includes at least one of the following: Basic geometric body structure, rotation direction of basic geometric body, view or acquisition device identifier or index corresponding to basic geometric body, rotation of basic geometric body, viewing direction in basic geometric body, combination method of basic geometric body, combination method of complex combined body, three-dimensional viewing space identifier.

11. The method according to claim 1, characterized in that: The 3D viewing space processing descriptor of the immersive media includes at least one of the following: The device type of 3D viewing space processing, the application type of 3D viewing space processing, the 3D viewing space processing method, and the identification of the 3D viewing space.

12. A device for processing immersive media data, characterized in that: include: an acquisition unit, configured to acquire immersive media data corresponding to a user's current viewing posture, wherein a three-dimensional viewing space is used to limit the rendering of the content viewed by the user, the three-dimensional viewing space being a range within which the user is allowed to move in the immersive media scene; a rendering unit, configured to determine a 3D viewing space timed metadata track and a 3D viewing space processing timed metadata track describing processing information of the 3D viewing space in a media file, wherein the 3D viewing space timed metadata track indicates a dynamically changing 3D viewing space of the immersive media data, and the 3D viewing space processing timed metadata track indicates a method for processing the dynamically changing 3D viewing space of the immersive media data; and render the immersive media data corresponding to the current viewing posture, wherein, when the current viewing posture has moved or is moving outside the 3D viewing space, the immersive media data is rendered according to the dynamically changing 3D viewing space of the immersive media data indicated by the 3D viewing space timed metadata track and the method for processing the dynamically changing 3D viewing space of the immersive media data indicated by the 3D viewing space processing timed metadata track; In which, the acquisition unit is further used to determine a media presentation description file when it is determined that the user's current viewing posture is moving or has moved outside the three-dimensional viewing space, wherein the media presentation description file includes a three-dimensional viewing space descriptor and / or a three-dimensional viewing space processing descriptor indicating the viewing of immersive media; and request the immersive media data corresponding to the current viewing posture according to the three-dimensional viewing space descriptor and / or the three-dimensional viewing space processing descriptor corresponding to the user's current viewing posture; the three-dimensional viewing space descriptor is used to indicate the three-dimensional viewing space, and the three-dimensional viewing space processing descriptor is used to indicate the processing method of the three-dimensional viewing space.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 11 when executed.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 11.

Images