A data processing method, apparatus, device, and storage medium for immersive media.
By adding an indication of the viewing content range to the presentation information of free-viewpoint videos, the problem of incomplete indication in the prior art is solved, and the effective viewing range indication and viewing effect of free-viewpoint videos are improved.
Patent Information
- Application Number
- CN202210139135.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-15
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-02-15
AI Technical Summary
The existing technology for presenting free-viewpoint videos has the problem of incomplete indication information, which cannot effectively indicate the viewing content range of free-viewpoint videos.
By adding an indication of the viewing content range to the presentation indication information of the free-viewpoint video, a media file of the free-viewpoint video is generated and packaged, and corresponding metadata information is added to the media file, including an indication of the viewing content range of the free-viewpoint video and scene boundary rotation range information that supports streaming transmission.
It effectively indicates the viewing range of free-viewpoint videos, guiding viewers to adjust their viewing angle reasonably within the indicated range to obtain a better viewing experience.
Smart Images

Figure CN116643644B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to the field of virtual reality (VR) technology, and more particularly to a data processing method for immersive media, a data processing device for immersive media, a computer device, and a computer-readable storage medium. Background Technology
[0002] Free-viewpoint video is an immersive media generated from image data captured by one or more cameras, containing different perspectives and supporting multi-degree-of-freedom interaction for the viewer. The rendering of free-viewpoint video requires rendering the camera-captured image data according to the presentation instructions; however, in practice, it has been found that the presentation instructions for free-viewpoint video in existing technologies are incomplete. Summary of the Invention
[0003] This application provides a data processing method, apparatus, device, and storage medium for immersive media, which can indicate the viewing content range of free-viewpoint video.
[0004] On one hand, embodiments of this application provide a data processing method for immersive media, including:
[0005] Acquire the media file of the immersive media, which contains presentation instructions for the free-viewpoint video, including instructions on the viewing range of the free-viewpoint video.
[0006] Display the free-view video according to the presentation instructions.
[0007] On one hand, embodiments of this application provide a data processing method for immersive media, including:
[0008] Acquire image data from one or more cameras and encode the image data into free-viewpoint video;
[0009] Add presentation indication information to free-viewpoint videos based on their application format; the presentation indication information for free-viewpoint videos includes indications of the viewing content range of the free-viewpoint video.
[0010] Encapsulate free-viewpoint video and free-viewpoint video presentation instructions into immersive media files.
[0011] On one hand, embodiments of this application provide a data processing apparatus for immersive media, including:
[0012] The acquisition unit is used to acquire the media file of the immersive media. The media file contains presentation instruction information for the free-view video, including the indication information of the viewing content range of the free-view video.
[0013] The processing unit is used to display the free-view video according to the presentation instructions of the free-view video.
[0014] On one hand, embodiments of this application provide a data processing apparatus for immersive media, including:
[0015] The acquisition unit is used to acquire image data from one or more cameras and encode the image data into free-viewpoint video.
[0016] The processing unit is used to add presentation indication information to the free-view video according to the application format of the free-view video; the presentation indication information of the free-view video includes indication information of the viewing content range of the free-view video;
[0017] And media files used to encapsulate free-viewpoint video and free-viewpoint video presentation instructions into immersive media.
[0018] Accordingly, this application provides a computer device, the device comprising:
[0019] A processor is used to load and execute computer programs;
[0020] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the data processing method for the immersive media described above.
[0021] Accordingly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method for immersive media.
[0022] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned immersive media data processing method.
[0023] In this embodiment, an immersive media file is obtained. This media file contains presentation instruction information for a free-viewpoint video. The presentation instruction information includes indications of the viewing content range of the free-viewpoint video. The free-viewpoint video is displayed according to these presentation instruction information. Therefore, by adding indications of the viewing content range of the free-viewpoint video to the presentation instruction information, the viewing content range of the free-viewpoint video can be indicated, guiding viewers to reasonably adjust their viewing angle within the indicated range to obtain a better viewing experience. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1a A schematic diagram of 6DoF provided in an exemplary embodiment of this application is shown;
[0026] Figure 1b A schematic diagram of 3DoF provided in an exemplary embodiment of this application is shown;
[0027] Figure 1c A schematic diagram of 3DoF+ provided in an exemplary embodiment of this application is shown;
[0028] Figure 1d This illustration shows a schematic diagram of an immersive media process from acquisition to consumption, provided by an exemplary embodiment of this application.
[0029] Figure 1e This illustration shows a schematic diagram of a free-viewpoint video data representation method provided by an exemplary embodiment of this application;
[0030] Figure 2 A flowchart illustrating a data processing method for immersive media provided in an exemplary embodiment of this application is shown;
[0031] Figure 3 A flowchart illustrating another data processing method for immersive media provided by an exemplary embodiment of this application is shown;
[0032] Figure 4 This invention provides a schematic diagram of the structure of a data processing apparatus for immersive media according to an exemplary embodiment of this application.
[0033] Figure 5 This invention provides a schematic diagram of the structure of another immersive media data processing apparatus according to an exemplary embodiment of the present application.
[0034] Figure 6 This illustration shows a structural schematic diagram of a content consumption device provided in an exemplary embodiment of this application;
[0035] Figure 7 A schematic diagram of the structure of a content creation device provided in an exemplary embodiment of this application is shown. Detailed Implementation
[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0037] The following describes some technical terms used in the embodiments of this application:
[0038] I. Immersive Media:
[0039] Immersive media refers to media files that provide immersive content, allowing viewers to experience visual, auditory, and other sensory sensations reminiscent of the real world. Based on the degree of freedom viewers have when consuming the media content, immersive media can be categorized as: 6DoF (Degree of Freedom) immersive media, 3DoF immersive media, and 3DoF+ immersive media.
[0040] II. Free-viewpoint video:
[0041] Free-viewpoint video, also known as multi-viewpoint video, is an immersive media generated from image data captured by multiple cameras, containing different perspectives and supporting multi-degree-of-freedom interaction for the viewer. For example, if a free-viewpoint video supports 3DoF+ interaction, then it is a 3DoF+ immersive media. Similarly, if a free-viewpoint video supports 6DoF interaction, then it is a 6DoF immersive media.
[0042] III. Track:
[0043] A track is a collection of media data during the media file encapsulation process. A media file can consist of one or more tracks. For example, a media file can typically contain a video track, an audio track, and a subtitle track.
[0044] IV. Sample:
[0045] A sample is a unit of encapsulation in the media file encapsulation process. A track consists of many samples. For example, a video track can consist of many samples. A sample is usually a video frame.
[0046] 5. DoF (Degree of Freedom, degree of freedom):
[0047] In this application, DoF refers to the degrees of freedom that a viewer has to move and interact with content when watching immersive media (such as free-viewpoint video), which can include 3DoF (three degrees of freedom), 3DoF+, and 6DoF (six degrees of freedom). 3DoF refers to the three degrees of freedom for the viewer's head to rotate around the x, y, and z axes. 3DoF+, in addition to the three degrees of freedom, also grants the viewer limited freedom of movement along the x, y, and z axes. 6DoF, in addition to the three degrees of freedom, also grants the viewer free freedom of movement along the x, y, and z axes.
[0048] 6. ISOBMFF (ISO Based Media File Format): This is a media file encapsulation standard. A typical ISOBMFF file is an MP4 file.
[0049] 7. DASH (Dynamic Adaptive Streaming over HTTP): This is an adaptive bitrate technology that enables high-quality streaming media to be delivered over the Internet through traditional HTTP web servers.
[0050] 8. MPD (Media Presentation Description, in DASH) is used to describe media segment information in a media file.
[0051] 9. Representation: refers to a combination of one or more media components in DASH. For example, a video file of a certain resolution can be regarded as a representation.
[0052] 10. Adaptation Sets: These are collections of one or more video streams in DASH. An Adaptation Set can contain multiple representations.
[0053] This application relates to immersive media data processing technology. Some concepts in the immersive media data processing process will be introduced below. In particular, the following embodiments of this application will use free-viewpoint video as an example for immersive media.
[0054] Figure 1a This illustration shows a schematic diagram of 6DoF provided in an exemplary embodiment of this application. 6DoF is divided into window 6DoF, omnidirectional 6DoF, and 6DoF. Window 6DoF means that the viewer of the immersive media has limited rotational movement along the X and Y axes, and limited translation along the Z axis; for example, the viewer cannot see the scene outside the window frame, and cannot pass through the window. Omnidirectional 6DoF means that the viewer of the immersive media has limited rotational movement along the X, Y, and Z axes; for example, the viewer cannot freely move through the 3D 360-degree VR content within the restricted movement area. 6DoF means that the viewer of the immersive media can freely translate along the X, Y, and Z axes; for example, the viewer can freely walk within the 3D 360-degree VR content. Similar to 6DoF are 3DoF and 3DoF+ production techniques. Figure 1b A schematic diagram of 3DoF provided in an exemplary embodiment of this application is shown; as follows: Figure 1b As shown, 3DoF refers to the viewer of immersive media being fixed at the center point in a three-dimensional space, while the viewer's head rotates along the X, Y, and Z axes to view the images provided by the media content. Figure 1c A schematic diagram of 3DoF+ provided in an exemplary embodiment of this application is shown, such as... Figure 1c As shown, 3DoF+ refers to the ability of immersive media viewers to move their heads within a limited space based on 3DoF to view the images provided by the media content when the virtual scene provided by the immersive media has a certain depth information.
[0055] Figure 1d This illustration shows a schematic diagram of an immersive media process from acquisition to consumption, provided by an exemplary embodiment of this application; as shown... Figure 1d As shown, the process of immersive media from acquisition to consumption includes:
[0056] (1) Video capture: Free-viewpoint video is usually captured by a camera array consisting of multiple cameras from multiple angles to capture the same three-dimensional scene, forming the scene's texture information (color information, etc.) and depth information (spatial distance information, etc.). The content production equipment can construct the immersive media consumed by the viewer on the immersive media side (such as 3DoF immersive media, 3DoF+ immersive media, or 6DoF immersive media, etc.) based on the position information of each virtual viewpoint in the free-viewpoint video and the texture information and depth information from different cameras.
[0057] (2) After obtaining the immersive media, the content production equipment compresses and encodes the immersive media to obtain a free-viewpoint video; for example, the content production equipment can compress and encode the immersive media using AVS3 encoding technology, HEVC encoding technology, etc., to obtain a free-viewpoint video.
[0058] (3) After obtaining the free-viewpoint video, the content production device encapsulates the data stream in the free-viewpoint video. Specifically, the content production device encapsulates the audio and video streams in a file container according to the immersive media file format (such as ISOBMFF (ISOBase Media File Format)) to form an immersive media file resource. This media file resource can be a media file or a media segment forming an immersive media file. The device also records the metadata of the immersive media file resource using Media Presentation Description (MPD) according to the immersive media file format requirements. Here, metadata is a general term for information related to the presentation of immersive media. This metadata may include descriptive information about the media content, descriptive information about the window, and signaling information related to the presentation of the media content, etc.
[0059] (4) The content production device transmits the free-viewpoint video encapsulation file to the content consumption device. This transmission process can be based on various transmission protocols, including but not limited to: DASH (Dynamic Adaptive Streaming over HTTP), HLS (HTTP Live Streaming), SMTP (Smart Media Transport Protocol), TCP (Transmission Control Protocol), etc.
[0060] (5) After obtaining the free-viewpoint video encapsulation file provided by the content production device, the content consumption device decapsulates the free-viewpoint video encapsulation file. The file decapsulation process on the content consumption device side is the reverse of the file encapsulation process on the content production device side. The content consumption device decapsulates the media file resources according to the file format requirements of immersive media to obtain audio and video streams.
[0061] (6) After the free-viewpoint video's encapsulated file is decapsulated, the content consumption device decodes the free-viewpoint video to obtain immersive media. The decoding process of the video stream by the content consumption device includes the following: ① Decoding the video stream to obtain a planar projection image. ② Reconstructing the projection image based on the media presentation description information to convert it into a 3D image. Here, reconstruction refers to the process of reprojecting the two-dimensional projection image into 3D space.
[0062] (7) The content consumption device presents the corresponding immersive media based on the virtual perspective of the viewer. The content consumption device renders the audio content obtained from audio decoding and the 3D image obtained from video decoding based on metadata related to rendering and viewport in the media presentation description information. Once rendering is complete, the 3D image is played and output. Specifically, if 3DoF and 3DoF+ production techniques are used, the content consumption device mainly renders the 3D image based on the current viewpoint, parallax, and depth information. If 6DoF production techniques are used, the content consumption device mainly renders the 3D image within the viewport based on the current viewpoint. Here, viewpoint refers to the viewer's position in the immersive media; parallax refers to the difference in line of sight between the viewer's two eyes or due to motion; and viewport refers to the viewing area.
[0063] Content production equipment and content consumption equipment can together form an immersive media system. Content production equipment refers to the computer equipment used by the provider of immersive media (e.g., the content creator of immersive media). This computer equipment can be a terminal (such as a PC, a smart mobile device (such as a smartphone)) or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Content consumption equipment refers to the computer equipment used by the user of immersive media (e.g., the viewer of immersive media). This computer equipment can be a terminal (such as a PC, a smart mobile device (such as a smartphone), VR devices (such as VR headsets, VR glasses), smart home appliances, in-vehicle terminals, aircraft, etc.).
[0064] It is understood that the data processing technology for immersive media involved in this application can be implemented using cloud technology; for example, using a cloud server as a content production device. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.
[0065] In practical applications, free-viewpoint video data can be expressed in different ways. Figure 1e This illustration shows a schematic diagram of a free-viewpoint video data representation method provided by an exemplary embodiment of this application. For example... Figure 1eAs shown, in this application, the free-viewpoint video data consists of image data acquired by multiple cameras and corresponding free-viewpoint video content description metadata. That is, this application expresses free-viewpoint video data through image data acquired by multiple cameras and corresponding free-viewpoint video content description metadata. Among them, the image data acquired by multiple cameras includes: texture maps acquired by multiple cameras and depth maps corresponding to the texture maps of multiple cameras.
[0066] Under the aforementioned free-viewpoint video data expression method, the process of immersive media involved in this application from acquisition to consumption is as follows:
[0067] (1) For content production equipment, firstly, the texture map and depth map acquired by the multi-camera matrix array are obtained; then the texture map information and the corresponding depth map information acquired by the multi-camera are stitched together; it should be noted that if the free-view video needs to consume background information in the subsequent consumption process, the background information should also be stitched into the image frame; then the planar video compression method is used to encode the stitched multi-camera depth map and texture map information; and the metadata description information of the above process is encapsulated into the video file.
[0068] (2) For content consumption devices, the video file is first decapsulated; then, based on the metadata information, the depth map and texture map information of each camera are decoded from the video file; then, based on the algorithm of the free-view video application, the depth map information and texture map information are combined to synthesize the free-view virtual viewpoint requested by the viewer of the immersive media.
[0069] In practical applications, content creation devices can use data boxes to guide content consumption devices in consuming immersive media files. A data box refers to a data block or object that includes metadata; that is, the data box contains the metadata of the corresponding media content. Immersive media can include multiple data boxes, such as rotation data boxes, overlay information data boxes, media file format data boxes, etc. The presentation indication information for free-viewpoint video can be configured in the media format data box of the immersive media, for example, in the free-viewpoint data box. In streaming, the description information corresponding to the presentation indication information can be configured in the transmission signaling file of the immersive media, for example, in the free-viewpoint camera descriptor. The presentation indication information of immersive media can be configured in the free-viewpoint data box (AvsFreeViewInfoBox), content description information, and the free-viewpoint boundary data box (AvsFreeViewBoundaryBox). According to the encoding standard of immersive media (such as AVS), the syntax of the free-viewpoint data box (AvsFreeViewInfoBox) of immersive media can be seen in Table 1 below:
[0070] Table 1
[0071]
[0072]
[0073]
[0074] The semantics of the syntax shown in Table 1 above are as follows:
[0075] In the AvsFreeViewInfoBox, codec_independency indicates the encoding and decoding independence between the texture maps and depth maps corresponding to each camera within the track. See Table 2 for specific indication methods.
[0076] Table 2
[0077]
[0078]
[0079] `depth_padding_size` indicates the width of the guard band in the depth map. `texture_padding_size` indicates the width of the guard band in the texture map. `camera_count` indicates the number of cameras that acquired image data from free-viewpoint video. `ExtCameraInfoStruct()` indicates the extrinsic parameters of the camera. `IntCameraInfoStruct()` indicates the intrinsic parameters of the camera. `camera_resolution_x` indicates the resolution width of the texture and depth maps acquired by the camera, and `camera_resolution_y` indicates the resolution height of the texture and depth maps acquired by the camera. `depth_downsample_factor` indicates the downsampling factor of the depth map; the actual resolution width and height of the depth map are half the resolution width and height acquired by the camera. depth_downsample_factor .
[0080] `depth_vetex_x` indicates the x-component of the offset of the top-left vertex of the depth map relative to the origin of the plane frame (the top-left vertex of the plane frame), and `depth_vetex_y` indicates the y-component of the offset of the top-left vertex of the depth map relative to the origin of the plane frame (the top-left vertex of the plane frame). `texture_vetex_x` indicates the x-component of the offset of the top-left vertex of the texture map relative to the origin of the plane frame (the top-left vertex of the plane frame), and `texture_vetex_y` indicates the y-component of the offset of the top-left vertex of the texture map relative to the origin of the plane frame (the top-left vertex of the plane frame).
[0081] In the camera extrinsic information structure (ExtCameraInfoStruct), `camera_pos_present` indicates whether camera position parameters are represented. A value of 1 indicates the presence of camera position parameters in the media file; a value of 0 indicates the absence of camera position parameters in the media file. `camera_ori_present` indicates whether camera orientation parameters are represented. A value of 1 indicates the presence of camera orientation parameters in the media file; a value of 0 indicates the absence of camera orientation parameters in the media file.
[0082] `camera_pos_x` indicates the x-coordinate of the camera position in the global reference coordinate system, expressed in meters; `camera_pos_y` indicates the y-coordinate of the camera position in the global reference coordinate system, expressed in meters; and `camera_pos_z` indicates the z-coordinate of the camera position in the global reference coordinate system, expressed in meters. The values of `camera_pos_x`, `camera_pos_y`, and `camera_pos_z` are incremented by 2. -16 The unit is meters.
[0083] `cam_quat_x` indicates the x-component of the camera rotation quaternion, `cam_quat_y` indicates the y-component, and `cam_quat_z` indicates the z-component. The values of `cam_quat_x`, `cam_quat_y`, and `cam_quat_z` are floating-point values in the range [-1, 1]. When any component of the rotation information is missing, its default value is 0. The calculation rules for each component are as follows:
[0084] qX=cam_quat_x÷2 30 qY = cam_quat_y ÷ 2 30 qZ=cam_quat_z÷2 30
[0085] The fourth component qW can be derived using the mathematical properties of quaternions:
[0086] qW = Sqrt(1–(qX) 2 +qY 2 +qZ 2 ))
[0087] The quaternion (w,x,y,z) represents the angle around the vector (x,y,z):
[0088] 2*cos^{-1}(w)=2*sin^{-1}(sqrt(x^{2}+y^{2}+z^{2})) is rotated.
[0089] In the camera intrinsic information structure (IntCameraInfoStruct), `camera_id` indicates the corresponding camera identifier. `camera_depth_present` indicates whether the camera depth parameter is represented. When the value of `camera_depth_present` is 1, it indicates that the camera depth parameter exists in the media file; when the value of this field is 0, it indicates that the camera depth parameter does not exist in the media file. `camera_type` indicates the camera's projection algorithm type. When the value of `camera_depth_present` is 0, it indicates that the projection algorithm type is ERP (Equi-Rectangular Projection); when the value of this field is 1, it indicates that the projection algorithm type is perspective projection; when the value of this field is 2, it indicates that the projection algorithm type is orthographic projection; when the value of this field is 3, it indicates that the projection algorithm type is a free-viewpoint pinhole camera model; when the value of this field is 4, it indicates that the projection algorithm type is a free-viewpoint fisheye camera model; other values are reserved.
[0090] Among them, erp_horizontal_fov is used to indicate the horizontal longitude range of the viewport area when projected via ERP, in radians. The value range of the erp_horizontal_fov field is (0, 2π); erp_vertical_fov is used to indicate the vertical dimension range of the viewport area when projected via ERP, in radians. The value range of the erp_horizontal_fov field is (0, π).
[0091] The `perspective_horizontal_fov` field indicates the horizontal range of the viewport area when projected using perspective, in radians. Its value range is (0, π). The `perspective_aspect_ratio` field indicates the aspect ratio (horizontal / vertical) of the viewport when projected using perspective. Its value is a 32-bit floating-point number, and the parsing process follows the IEEE 754 standard.
[0092] `ortho_horizontal_size` indicates the horizontal size of the viewport when projected orthogonally, in meters. This field is a 32-bit floating-point number, and its resolution follows the IEEE 754 standard. `ortho_aspect_ratio` indicates the aspect ratio (horizontal / vertical) of the viewport when projected orthogonally. This field is also a 32-bit floating-point number, and its resolution follows the IEEE 754 standard.
[0093] camera_focal_length_x is used to indicate the x-component of the camera focal length in a free-view pinhole camera model, and camera_focal_length_y is used to indicate the y-component of the camera focal length in a free-view pinhole camera model.
[0094] camera_principle_point_x is used to indicate the x-component of the camera principal optical axis offset in the image coordinate system of the free-view fisheye camera model, and camera_principle_point_y is used to indicate the y-component of the camera principal optical axis offset in the image coordinate system of the free-view fisheye camera model.
[0095] `camera_near_depth` indicates the near-plane depth (or distance) of the camera's associated frustum, and `camera_far_depth` indicates the far-plane depth (or distance) of the camera's associated frustum; the values of `camera_near_depth` and `camera_far_depth` are incremented by 2. -16 The unit is meters.
[0096] Based on the encoding standards for immersive media (such as AVS), the syntax for the content description information of immersive media can be found in Table 3 below:
[0097] Table 3
[0098]
[0099]
[0100] Table 3 above mentions 3DPoint: x, y, z, which represent the x, z, and y coordinates of a 3D point in the Cartesian coordinate system, respectively; the semantics of the syntax shown are as follows:
[0101] Cuboid RegionStruct: cuboid_dx indicates the size of the cuboid subregion in Cartesian coordinates relative to the anchor point along the x-axis, cuboid_dy indicates the size of the cuboid subregion in Cartesian coordinates relative to the anchor point along the y-axis, and cuboid_dz indicates the size of the cuboid subregion in Cartesian coordinates relative to the anchor point along the z-axis.
[0102] SpheroidStruct: radius_x indicates the radius of the spherical sub-region in the x-axis of the Cartesian coordinate system, radius_y indicates the radius of the spherical sub-region in the y-axis of the Cartesian coordinate system, and radius_z indicates the radius of the spherical sub-region in the z-axis of the Cartesian coordinate system.
[0103] HalfspaceStruct: `normal_x` indicates the planar normal of the hemispherical sub-region in the x-axis of the Cartesian coordinate system; `normal_y` indicates the planar normal of the hemispherical sub-region in the y-axis of the Cartesian coordinate system; `normal_z` indicates the planar normal of the hemispherical sub-region in the z-axis of the Cartesian coordinate system. `distance` indicates the distance from the origin along the normal vector to the plane of the hemispherical structure.
[0104] CylinderStruct: radius_x indicates the radius of the cylindrical sub-region in the x-dimensional coordinate system, radius_y indicates the radius of the cylindrical sub-region in the x-dimensional coordinate system, and height indicates the height of the cylindrical sub-region in the z-dimensional coordinate system.
[0105] The AvsFreeViewBoundaryBox is used to indicate information about the boundaries of a free-view video scene. The syntax of the AvsFreeViewBoundaryBox for immersive media, based on the immersive media coding standard (such as AVS), can be found in Table 4 below:
[0106] Table 4
[0107]
[0108]
[0109] The semantics of the AvsFreeViewBoundaryBox syntax shown in Table 4 above are as follows:
[0110] `boundary_camera_count` indicates the number of cameras constituting the boundary of the free-view video scene. `camera_id` indicates the identifier of the camera constituting the boundary of the free-view video scene; when the position, orientation, and viewport area of the boundary camera have already been declared in the AvsFreeViewInfoBox, the relevant information of the boundary camera can be determined only by the camera identifier. `boundary_space_present` indicates whether there is an additional scene rendering boundary. When the value of `boundary_space_present` is 0, the scene rendering boundary is determined by the parameters of the boundary camera, and the shooting range of the boundary camera constitutes the scene rendering boundary; when the value of this field is 1, the scene rendering boundary is determined by `boundary_space`, which indicates the range of the scene boundary. If the area corresponding to `boundary_space` is larger than the area formed by the boundary camera, the area outside the shooting range of the boundary camera is rendered by the content consumption device based on the content captured by the boundary camera. `bounding_space_type` indicates the type of scene boundary. The meanings of the values of the `bounding_space_type` field are shown in Table 5 below.
[0111] Table 5
[0112]
[0113]
[0114] The default_origin_point field is used to indicate the origin of the scene. When the default_origin_point field is 1, it means that the scene boundary uses the coordinate origin (0,0,0) as the origin of the scene. When the default_origin_point field is 0, it means that the scene boundary uses the specified point as the origin of the scene.
[0115] The `boundary_exceeded_switch` field is used to indicate the handling method when the viewer's viewing behavior in immersive media exceeds the scene boundary. When the `boundary_exceeded_switch` field is 0, a virtual viewport region based on the origin and oriented at (0,0,0) is rendered for the viewer. When the `boundary_exceeded_switch` field is 1, the content area captured by the camera corresponding to `dest_camera_id` is rendered for the viewer. When the `boundary_exceeded_switch` field is 2, a virtual viewport region based on the position and orientation specified by `dest_virtual_camera` is rendered for the viewer.
[0116] As can be seen from Tables 1, 3, and 4 above, while the presentation indication information for free-viewpoint videos indicates parameters related to the camera used for shooting the video and information related to scene boundaries, it fails to clearly and effectively indicate the viewing content range of the immersive media. To address this issue, this application has made corresponding improvements and proposes a new data processing method for immersive media in specific applications of free-viewpoint videos. Using the technology proposed in this invention, the viewing content range of free-viewpoint videos can be indicated, guiding viewers to watch the videos reasonably within the indicated range and obtain a better viewing experience.
[0117] The data processing flow for immersive media in this application is as follows:
[0118] For content production equipment, on the one hand, the content production equipment generates a free-viewpoint video bitstream and encapsulates the video bitstream into a media file. Based on the application form of the free-viewpoint video, corresponding metadata information is added to the free-viewpoint video file, which may specifically include indication information of the viewing content range of the free-viewpoint video.
[0119] On the other hand, if the content production device supports streaming, it slices the free-viewpoint video into media segments suitable for streaming (according to the specifications of the transmission standard) and generates corresponding transmission signaling. This transmission signaling can be used to indicate the rotation range information of the scene boundaries. After generating the corresponding transmission signaling, the content production device transmits the signaling file to the content consumption device.
[0120] For content consumption devices, depending on the application, they can download the complete immersive media file and play it locally; or they can establish streaming transmission with content production devices and adaptively select the appropriate free-viewpoint video stream for consumption based on the transmission signaling.
[0121] To support the above implementation process, this application adds several descriptive fields at the system layer. The following example, using extended ISOBMFF data boxes and DASH signaling, defines relevant fields to support free-view video applications; the extensions to the free-view data box (AvsFreeViewInfoBox) are shown in Table 6 below:
[0122] Table 6
[0123]
[0124]
[0125] The semantics of the extended syntax added to Table 6 compared to Table 1 are as follows: The duplicated_camera_params field indicates that when duplicated_camera_params is 1, it means that the camera internal information structure IntCameraInfoStruct and the camera frame information structure CameraFrameInfoStruct are identical for each camera in the AvsFreeViewInfoBox, indicating them only once, and these parameters apply to all cameras; in this case, the camera_id field in IntCameraInfoStruct is meaningless; the camera frame information structure CameraFrameInfoStruct can specifically include the following fields: camera_resolution_x, camera_resolution_y, depth_downsample_factor, texture_padding_size, and depth_padding_size. When duplicated_camera_params is 0, it means that the cameras indicated in the AvsFreeViewInfoBox have different camera-related parameters.
[0126] The extensions to the AvsFreeViewAngleRangeBox are shown in Table 7 below:
[0127] Table 7
[0128]
[0129]
[0130] The semantics of the AvsFreeViewAngleRangeBox syntax shown in Table 7 above are as follows:
[0131] The yaw range_flag field indicates the virtual yaw angle of the immersive media viewer. When yaw_range_flag is set to a first value (e.g., 1), it means that the yaw angle of the immersive media viewer's virtual perspective is limited by the yaw angle range indicated by the yaw angle range fields (rotation_yaw_min, rotation_yaw_max). Rotation_yaw_min indicates the minimum yaw angle for the immersive media viewer, and rotation_yaw_max indicates the maximum yaw angle for the immersive media viewer. The rotation_yaw_min and rotation_yaw_max values are in increments of 2. -16 For precision, the value range is [-180°*2]. 16 180°*2 16 –1]. The limitation here refers to the fact that the yaw angle of the virtual viewpoint of the viewer of the immersive media must change within the range of the yaw angle indicated by the yaw angle range field. When yaw_range_flag is set to the second setting value (such as 0), it means that the yaw angle of the virtual viewpoint of the viewer of the immersive media is not limited, and the range is [-180°, 180°), that is, the yaw angle of the virtual viewpoint of the viewer of the immersive media can change without restriction within the range of [-180°, 180°).
[0132] The pitch angle flag field `pitch_range_flag` indicates the virtual viewing angle pitch angle for the viewer of the immersive media. When `pitch_range_flag` is set to the first preset value (e.g., 1), it means that the virtual viewing angle pitch angle for the viewer is limited by the pitch angle range indicated by the pitch angle range fields (`rotation_pitch_min`, `rotation_pitch_max`). `rotation_pitch_min` indicates the minimum pitch angle for the viewer, and `rotation_pitch_max` indicates the maximum pitch angle. The `rotation_pitch_min` and `rotation_pitch_max` values are in increments of 2. -16 For precision, the value range is [-90°*2]. 16 90°*2 16The limitation here refers to the fact that the pitch angle of the virtual viewpoint of the viewer of the immersive media must be within the pitch angle range indicated by the pitch angle range field. When pitch_range_flag is set to the second setting value (such as 0), it means that the pitch angle of the virtual viewpoint of the viewer of the immersive media is not limited, and the range is [-90°, 90°]. That is to say, the pitch angle of the virtual viewpoint of the viewer of the immersive media can change without restriction within the range of [-90°, 90°].
[0133] The roll range_flag field indicates the roll angle of the virtual viewpoint for the viewer of the immersive media. When roll_range_flag is set to the first preset value (e.g., 1), it means that the roll angle of the virtual viewpoint for the viewer of the immersive media is limited by the roll angle range indicated by the roll angle range fields (rotation_roll_min, rotation_roll_max). Rotation_roll_min indicates the minimum roll angle for the viewer of the immersive media, and rotation_roll_max indicates the maximum roll angle for the viewer of the immersive media. The rotation_roll_min and rotation_roll_max are in increments of 2. -16 For precision, the value range is [-180°*2]. 16 180°*2 16 –1]. The limitation here refers to the fact that the roll angle of the virtual viewpoint of the viewer of the immersive media must be within the roll angle range indicated by the roll angle range field. When roll_range_flag is set to the second setting value (such as 0), it means that the roll angle of the virtual viewpoint of the viewer of the immersive media is not limited, and the range is [-180°, 180°), that is, the roll angle of the virtual viewpoint of the viewer of the immersive media can change without restriction within the range of [-180°, 180°).
[0134] The depth range flag field `depth_range_flag` indicates the virtual viewing angle depth for viewers of immersive media. When `depth_range_flag` is set to a first value (e.g., 1), it means that the virtual viewing angle depth range for viewers of immersive media is limited by the depth range fields (`depth_min`, `depth_max`). Here, `depth_min` indicates the minimum virtual viewing angle depth for viewers of immersive media, and `depth_max` indicates the maximum virtual viewing angle depth for viewers of immersive media. The `depth_min` and `depth_max` values are in increments of 2. -16For precision. The limitation here refers to the fact that the range of changes in the virtual viewing depth for viewers of immersive media must be within the depth range indicated by the depth range field. When depth_range_flag is set to the second setting (e.g., 0), it means that the virtual viewing depth for viewers of immersive media is not limited.
[0135] In another implementation, the extension of the free view range data box (AvsFreeViewAngleRangeBox) is shown in Table 8 below:
[0136] Table 8
[0137]
[0138] As shown in Table 8 above, the AvsFreeViewAngleRangeBox contains a 3D rotation quaternion, which indicates the 3D rotation range of the virtual viewpoint of the viewer in the immersive media. This 3D rotation quaternion includes x, y, and z components. The AvsFreeViewAngleRangeBox contains range fields for the x, y, and z components of the quaternion; and the semantics of the syntax are as follows:
[0139] The range field for the x-component of a quaternion indicates the range of values for the x-component in the 3D rotated quaternion. This field includes `rotation_x_min` and `rotation_x_max`; where `rotation_x_min` indicates the minimum value of the x-component in the 3D rotated quaternion (qX, qY, qZ, qW), and `rotation_x_max` indicates the maximum value of the x-component in the 3D rotated quaternion (qX, qY, qZ, qW). Similarly, the range field for the y-component of a quaternion indicates the range of values for the y-component in the 3D rotated quaternion. This field includes `rotation_y_min` and `rotation_y_may`; where `rotation_y_min` indicates the minimum value of the y-component in the 3D rotated quaternion (qY, qY, qZ, qW), and `rotation_y_may` indicates the maximum value of the y-component in the 3D rotated quaternion (qY, qY, qZ, qW). The range field for the z-component of a quaternion indicates the range of values for the z-component in a 3D rotated quaternion. This range field includes `rotation_z_min` and `rotation_z_maz`; `rotation_z_min` indicates the minimum value of the z-component in the 3D rotated quaternion (qZ,qY,qZ,qW), and `rotation_z_maz` indicates the maximum value of the z-component in the 3D rotated quaternion (qZ,qY,qZ,qW). The range fields for the x-component, y-component, and z-component of the quaternion are set to a 2... -30 The unit is [-2], and the range of values is [-2]. 30 ,2 30 The w component in a quaternion can be derived from the other three components:
[0140] qW = Sqrt(1–(qX) 2 +qY 2 +qZ 2 ))
[0141] The semantics of the depth range flag field depth_range_flag, as well as the depth range fields depth_min and depth_max, can be found in Table 7 above, and will not be repeated here.
[0142] In one embodiment, the AvsFreeViewAngleRangeBox is an optional data box. When it is included in the corresponding free-view video track, it indicates that the viewer's viewing angle for the immersive media is limited. Conversely, if it is not included, the viewer's viewing angle for the immersive media is considered unrestricted.
[0143] In another embodiment, the AvsFreeViewAngleRangeBox can also be included in the AvsFreeViewBoundaryBox, and the angle range flag field angle_range_flag is used to control whether the free-view video is presented according to the instructions in the AvsFreeViewAngleRangeBox, thereby indicating whether the viewing angle of the immersive media of the free-view video is limited.
[0144] In another implementation, besides expressing the limited range of the viewing content from the perspective of the viewer's viewing behavior, the limited range of the viewing content can also be expressed by combining information about the scene boundaries of the content itself. The extensions to the AvsFreeViewAngleRangeBox are shown in Table 9 below:
[0145] Table 9
[0146]
[0147]
[0148] The semantics of the AvsFreeViewAngleRangeBox syntax shown in Table 9 above are as follows:
[0149] The `vector_origin_point` field indicates the coordinates of the center point of the bottom surface of the boundary region. The `vector_end_point` field indicates the coordinates of the target point on the bottom surface of the boundary region, which can be any point on the edge of the bottom surface of the boundary region. The `angle_yaw_range` field indicates the viewing angle range of the immersive media viewer. Specifically, using the vector from the coordinate point indicated by the `vector_origin_point` field to the coordinate point indicated by the `vector_end_point` field as the starting vector, rotating clockwise within the angle range specified by the `angle_yaw_range` field, the resulting portion of the boundary region represents the actual free-view content boundary range that the immersive media viewer can consume.
[0150] The height_range_flag field indicates the viewpoint height of the immersive media viewer. When the height_range_flag field is set to the first setting (e.g., 1), it means that the actual free-viewpoint content boundary that the immersive media viewer can consume is limited in height by the height range indicated by the height_range field. The height_range field indicates the maximum height of the virtual viewpoint for the viewer during viewing, and the minimum height is the height of the bottom of the current boundary area. When the height_range_flag field is set to the second setting (e.g., 0), it means that the actual free-viewpoint content boundary that the immersive media viewer can consume is not limited within the height range indicated by the 3DBoundingBoxStruct (see Table 3).
[0151] The extensions to the AvsFreeViewBoundaryBox are shown in Table 10 below:
[0152] Table 10
[0153]
[0154]
[0155] The semantics of the AvsFreeViewBoundaryBox syntax shown in Table 10 above are as follows:
[0156] The AvsFreeViewBoundaryInfoStruct is used to indicate scene boundary information. When the angle_range_flag field is set to the first preset value (e.g., 1), it indicates that the viewer's viewing angle for the free-view video is limited; when the angle_range_flag field is set to the second preset value (e.g., 0), it indicates that the viewer's viewing angle for the free-view video is unrestricted. The AvsFreeViewAngleRangeBox is used to indicate the range of restricted viewing angles for the immersive media viewer, as detailed in Tables 7-9 above, and will not be elaborated upon here.
[0157] The description information corresponding to the free-view data box (AvsFreeViewInfoBox) is stored in the free-view camera descriptor (AvsFreeViewCamInfo) provided in this application embodiment. The free-view camera descriptor (AvsFreeViewCamInfo) is a supplemental property element, and its @schemeIdUri attribute is "urn:avs:ims:2018:av3f". The free-view camera descriptor (AvsFreeViewCamInfo) is included in the transmission signaling file, which is encapsulated in the representation layer of the immersive media's media presentation description file, or in the adaptive set layer; an adaptive set layer can contain one or more representation layers. When the description information exists in the target adaptive set layer, it is used to describe all representation layers in the target adaptive set layer; when the description information exists in the target representation layer, it is used to describe the target representation layer. The attributes contained in the transmission signaling file are shown in Table 11 below:
[0158] Table 11
[0159]
[0160]
[0161]
[0162] Where O represents the corresponding attribute as Optional; CM represents the corresponding attribute as Conditional Mandatory; and M represents the corresponding attribute as Mandatory. The expanded free-view camera descriptor (AvsFreeViewCamInfo) adds the following: AvsFreeViewCam@camera_quat_x, AvsFreeViewCam@camera_quat_y, AvsFreeViewCam@camera_quat_z, AvsFreeViewCam@camera_type, AvsFreeViewCam@erp_horizontal_fov, AvsFreeViewCam@erp_vertical_fov, AvsFreeViewCam@perspective_horizontal_fov, and AvsFreeViewCam@perspective_aspect_ratio. The following are descriptions of AvsFreeViewCam@ortho_aspect_ratio, AvsFreeViewCam@ortho_horizontal_size, AvsFreeViewCam@clipping_near_plane, AvsFreeViewCam@clipping_far_plane, AvsFreeViewCam@camera_principle_point_x, AvsFreeViewCam@camera_principle_point_y, AvsFreeViewCam@camera_group_id, AvsFreeViewCam@boundary_camera_flag, and related elements and attributes.
[0163] The description information corresponding to the AvsFreeViewAngleRangeBox is stored in the AvsFreeViewAngleRange descriptor provided in this embodiment of the application. The AvsFreeViewAngleRange is a supplemental property element with the @schemeIdUri attribute "urn:avs:ims:2018:asar". The AvsFreeViewAngleRange is included in the transmission signaling file, which is encapsulated in the representation layer of the immersive media's media presentation description file or in the adaptive set layer; an adaptive set layer can contain one or more representation layers. When the description information exists in the target adaptive set layer, it is used to describe all representation layers in the target adaptive set layer; when the description information exists in the target representation layer, it is used to describe the target representation layer. The attributes contained in the transmission signaling file are shown in Table 12 below:
[0164] Table 12
[0165]
[0166]
[0167] Where O represents the corresponding attribute as Optional. The expanded free-view video angle range descriptor (AvsFreeViewAngleRange) adds: AvsFreeViewAngleRange, AvsFreeViewAngleRange@rotation_yaw_min, AvsFreeViewAngleRange@rotation_yaw_max, AvsFreeViewAngleRange@rotation_pitch_min, AvsFreeViewAngleRange@rotation_pitch_max, AvsFreeViewAngleRange@rotation_roll_min, AvsFreeViewAngleRange@rotation_roll_max, AvsFreeViewAngleRange@depth_min, AvsFreeViewAngleRange@depth_max, and descriptions of these elements and attributes.
[0168] Based on the free-viewpoint data box and camera frame information structure shown in Table 6 of this application embodiment, and combined with the description of the free-viewpoint camera descriptor shown in Table 11, the indication method of camera parameters is optimized; based on the free-viewpoint range data box shown in Tables 7, 8 and 9 of this application embodiment, and combined with the description of the free-viewpoint video angle range descriptor shown in Table 12, the viewing content range of the free-viewpoint video is indicated; guiding the viewer to reasonably adjust the viewing angle within the indicated range to obtain a better viewing effect.
[0169] Figure 2 A flowchart illustrating a data processing method for immersive media provided in an exemplary embodiment of this application is shown; the method can be executed by a content consumption device in an immersive media system, and the method includes the following steps S201-S202:
[0170] S201. Obtain the media files for immersive media.
[0171] The media file contains presentation instructions for the free-viewpoint video, including instructions on the viewing content range of the free-viewpoint video. These instructions include: the supported viewing angle rotation range of the free-viewpoint video, and the information indicating the content to be viewed.
[0172] In this application, if the camera parameters of multiple cameras, IntCameraInfoStruct and CameraFrameInfoStruct (CameraFrameInfoStruct includes fields: camera_resolution_x, camera_resolution_y, depth_downsample_factor, texture_padding_size, depth_padding_size), are the same, then the camera indication information in the free-view data box can be declared only once.
[0173] In one implementation, the immersive media file is sliced into multiple media segments. The content consumption device acquires the transmission signaling file of the immersive media, which contains descriptive information corresponding to the presentation indication information of the free-viewpoint video. After acquiring the transmission signaling file of the immersive media, the content consumption device acquires the immersive media file based on the transmission signaling file. Specifically, the content consumption device can determine the media segment corresponding to the image data required by the viewer of the immersive media based on the virtual perspective of the viewer and the descriptive information in the transmission signaling file, and retrieve the determined media segment through streaming.
[0174] In one embodiment, the transmission signaling file includes at least one of the following: a free-view camera descriptor and a free-view video angle range descriptor; wherein, the free-view camera descriptor is used to indicate the parameters of each camera in one or more camera groups (such as projection method, camera position, etc.); and the free-view video angle range descriptor is used to indicate the boundary information of the free-view scene in the free-view video.
[0175] In another embodiment, the transmission signaling file includes a representation layer and an adaptive set layer, wherein the adaptive set layer includes one or more representation layers; when description information exists in the target adaptive set layer, it is used to describe all representation layers in the target adaptive set layer; when description information exists in the target representation layer, it is used to describe the target representation layer; wherein the description information may include one or more of a free-view camera descriptor and a free-view angle range descriptor.
[0176] S202. Display the free-view video according to the presentation instructions for the free-view video.
[0177] In one implementation, the indication information for the viewing content range of the free-viewpoint video includes indication information for the viewing content of the free-viewpoint video; the viewing content of the free-viewpoint video is generated based on image data captured by one or more cameras; the presentation indication information of the free-viewpoint video is metadata information, which further includes a free-viewpoint data box, and the free-viewpoint data box contains camera parameter indication information. The content consumption device displays the free-viewpoint video according to the presentation indication information, including: determining the image data required for viewing from the media file based on the virtual perspective of the viewer of the immersive media; and decoding and displaying the image data according to the camera parameter indication information.
[0178] In one embodiment, the free-viewpoint data box includes camera parameters and a duplicated camera parameter indication field, duplicated_camera_params; wherein, the camera parameters include the camera internal information structure (IntCameraInfoStruct) and the camera frame information structure (CameraFrameInfoStruct), as detailed in Tables 1 and 6 above, and will not be repeated here.
[0179] When the duplicated camera parameter indicator field takes the first set value (e.g., duplicated_camera_params = 1), it means that the camera parameters in the free-view data box are effective for all cameras. In this case, the IntCameraInfoStruct, camera_resolution_x, camera_resolution_y, depth_downsample_factor, texture_padding_size, and depth_padding_size are only indicated once in the free-view data box. When the duplicated camera parameter indicator field takes the second set value (e.g., duplicated_camera_params = 0), it means that different cameras in the free-view data box have their own camera parameters.
[0180] In another implementation, the presentation indication information for the free-viewpoint video includes indication information on the viewpoint rotation range supported by the free-viewpoint video. This presentation indication information is metadata information, which includes a free-viewpoint range data box (AvsFreeViewAngleRangeBox); the free-viewpoint range data box contains indication information on the viewpoint rotation range supported by the free-viewpoint video. The content consumption device displays the free-viewpoint video according to the presentation indication information, including: during the display of image data, the content consumption device limits the rotation range of the virtual viewpoint for the viewer of the immersive media according to the viewpoint rotation range indication information supported by the free-viewpoint video.
[0181] In one embodiment, the free-view range data box includes a yaw angle flag field yaw_range_flag, yaw angle range fields rotation_yaw_min and rotation_yaw_max, pitch angle flag field pitch_range_flag, pitch angle range fields rotation_pitch_min and rotation_pitch_max, roll angle flag field roll_range_flag and roll angle range fields rotation_roll_min and rotation_roll_max, depth range flag field depth_range_flag, and depth range fields depth_min and depth_max. The yaw angle flag field indicates the yaw angle of the virtual viewpoint for the viewer of the immersive media. When the yaw angle flag field is set to the first preset value (e.g., yaw_range_flag = 1), it means that the yaw angle of the virtual viewpoint for the viewer is limited to the yaw angle range indicated by the yaw angle range field (rotation_yaw_min, rotation_yaw_max). For example, if rotation_yaw_min = 10° and rotation_yaw_max = 90°, then the viewer's virtual viewpoint is limited to the yaw angle range [10°, 90°]. When the yaw angle flag field is set to the second preset value (e.g., yaw_range_flag = 0), it means that the yaw angle of the virtual viewpoint for the viewer is not limited, that is, the yaw angle of the virtual viewpoint for the viewer can vary freely within the range [-180°, 180°).
[0182] Similarly, the pitch angle flag field indicates the virtual viewing angle of the immersive media viewer. When the pitch angle flag field is set to the first setting value (e.g., pitch_range_flag = 1), it means that the virtual viewing angle of the immersive media viewer is limited to the pitch angle range indicated by the pitch angle range field (rotation_pitch_min, rotation_pitch_max). For example, if rotation_pitch_min = -20° and rotation_pitch_max = 80°, then the viewer's virtual viewing angle is limited to the pitch angle range [-20°, 80°]. When the pitch angle flag field is set to the second setting value (e.g., pitch_range_flag = 0), it means that the virtual viewing angle of the immersive media viewer is not limited, that is, the virtual viewing angle of the immersive media viewer can vary freely within the range [-90°, 90°].
[0183] Similarly, the roll angle flag field is used to indicate the roll angle of the virtual viewpoint of the viewer while watching immersive media. When the roll angle flag field is set to the first setting value (e.g., roll_range_flag = 1), it means that the roll angle of the virtual viewpoint of the viewer while watching immersive media is limited to the roll angle range indicated by the roll angle range field (rotation_roll_min, rotation_roll_max). For example, if rotation_roll_min = 20° and rotation_roll_max = 70°, then the viewer's virtual viewpoint is limited to the roll angle range [20°, 70°]. When the roll angle flag field is set to the second setting value (e.g., roll_range_flag = 0), it means that the roll angle of the virtual viewpoint of the viewer while watching immersive media is unrestricted, that is, the roll angle of the virtual viewpoint of the viewer while watching immersive media can vary freely within the range of [-180°, 180°).
[0184] Similarly, the depth range flag field indicates the virtual viewing angle depth of the immersive media viewer; the depth range field indicates the depth range to which the virtual viewing angle of the immersive media viewer belongs; when the depth range flag field is set to the first setting value (e.g., depth_range_flag = 1), it means that the virtual viewing angle depth of the immersive media viewer is limited by the depth range indicated by the depth range field (depth_min, depth_max); for example, depth_min = 0, depth_max = 100, then the viewer's virtual viewing angle is limited to the depth range [0, 100]. When the depth range flag field is set to the second setting value (e.g., depth_range_flag = 0), it means that the virtual viewing angle depth of the immersive media viewer is not limited.
[0185] In another embodiment, the AvsFreeViewAngleRangeBox contains a 3D rotation quaternion used to indicate the 3D rotation range of the virtual viewpoint of the viewer of the immersive media; this 3D rotation quaternion includes x-components, y-components, and z-components. The aforementioned AvsFreeViewAngleRangeBox contains range fields for the x-component, y-component, and z-component of the quaternion. Specifically, the range field for the x-component indicates the value range of the x-component in the 3D rotation quaternion, and includes rotation_x_min and rotation_x_max; the range field for the y-component indicates the value range of the y-component in the 3D rotation quaternion, and includes rotation_y_min and rotation_y_max; the range field for the z-component indicates the value range of the z-component in the 3D rotation quaternion, and includes rotation_z_min and rotation_z_max. For detailed implementation methods, please refer to Table 8 above, which will not be repeated here.
[0186] In another embodiment, besides expressing the limited range of the viewing content from the perspective of the viewer's viewing behavior, the limited range of the viewing content can also be expressed by combining information about the scene boundaries of the content itself. The AvsFreeViewAngleRangeBox contains a vector origin field (vector_origin_point), a vector end field (vector_end_point), and a yaw angle range field (angle_yaw_range); these fields are used to indicate the viewing angle range of the immersive media viewer. Specifically, using the vector from the coordinate point indicated by the vector origin field to the coordinate point indicated by the vector end field as the starting vector, rotating clockwise by the angle range specified by the yaw angle range field (angle_yaw_range), the resulting partial boundary area represents the actual free-view content boundary range that the immersive media viewer can consume.
[0187] In addition, the AvsFreeViewAngleRangeBox also includes a height_range_flag field to indicate the viewpoint height of the immersive media viewer. When the height_range_flag field is set to the first setting (e.g., 1), it means that the actual free-viewpoint content boundary that the immersive media viewer can consume is limited in height by the height range indicated by the height_range field. The height_range field indicates the maximum height of the virtual viewpoint of the immersive media viewer, and the minimum height is the height of the bottom of the current boundary area. When the height_range_flag field is set to the second setting (e.g., 0), it means that the actual free-viewpoint content boundary that the immersive media viewer can consume is not limited within the height range indicated by the 3DBoundingBoxStruct (see Table 3).
[0188] In one implementation, the free-viewpoint video is encapsulated in at least one track, which contains a target track containing image data of the free-viewpoint video. When the target track contains a free-viewpoint range data box, it indicates that the viewing angle corresponding to the image data in the target track is limited. When the target track does not contain a free-viewpoint range data box, it indicates that the viewing angle corresponding to the image data in the target track is unrestricted.
[0189] In another implementation, the metadata information includes a free-view data box (AvsFreeViewInfoBox) and a free-view range data box (AvsFreeViewAngleRangeBox) contained within the free-view data box. The free-view data box also contains an angle range flag field (angle_range_flag). When the angle range flag field is set to a first set value (e.g., angle_range_flag = 1), it indicates that the viewing angle of the image data corresponding to the free-view data box is limited, and the free-view range data box indicates the limited range of the viewing angle. When the angle range flag field is set to a second set value (e.g., angle_range_flag = 0), it indicates that the viewing angle of the image data corresponding to the free-view data box is not limited.
[0190] In another implementation, the metadata information includes a free-view boundary data box (AvsFreeViewBoundaryBox) and a free-view range data box (AvsFreeViewAngleRangeBox) contained within the free-view boundary data box. The free-view boundary data box also contains an angle range flag field (angle_range_flag). When the angle range flag field is set to a first preset value (e.g., angle_range_flag = 1), it indicates that the viewing angle of the image data corresponding to the free-view boundary data box is limited, and the free-view range data box indicates the limited range of the viewing angle. When the angle range flag field is set to a second preset value (e.g., angle_range_flag = 0), it indicates that the viewing angle of the image data corresponding to the free-view boundary data box is not limited.
[0191] In this embodiment, an immersive media file is obtained. This media file contains presentation instruction information for free-viewpoint video. The presentation instruction information includes indications of the viewing content range of the free-viewpoint video. The free-viewpoint video is displayed according to these presentation instruction information. It is evident that by adding indications of the viewing content range of the free-viewpoint video to the presentation instruction information, the viewing content range of the free-viewpoint video can be indicated, guiding viewers to reasonably adjust their viewing angle within the indicated range to obtain a better viewing experience. Furthermore, the method of indicating camera parameters is optimized by adding a repeated camera parameter indication field to the presentation instruction information.
[0192] Figure 3 A flowchart illustrating another data processing method for immersive media provided in an exemplary embodiment of this application is shown; the method can be executed by a content consumption device in an immersive media system, and the method includes the following steps S301-S303:
[0193] S301. Acquire image data from one or more cameras and encode the image data into free-viewpoint video.
[0194] For a detailed implementation of step S301, please refer to Figure 1d The implementation methods of medium-length video encoding will not be described in detail here.
[0195] S302. Add presentation instruction information to the free-view video according to its application format.
[0196] The presentation indication information for free-viewpoint videos includes information indicating the viewing content range. Content creation devices add presentation indication information to free-viewpoint videos based on their application format. This is the reverse process of content consumption devices displaying free-viewpoint videos based on this information. For details, please refer to [reference needed]. Figure 2 The process of displaying free-viewpoint video on a content consumption device based on the presentation instructions for the free-viewpoint video.
[0197] Taking the free-view range data box as an example, suppose the range information of the virtual view in a free-view scene in a free-view video is: yaw angle range [10°, 90°], pitch angle range [-20°, 80°], and roll angle range [20°, 70°]. Then, based on the above range information, the content production device configures the values of each field in the free-view range data box corresponding to the free-view scene as follows: yaw_range_flag = 1, rotation_yaw_min = 10°, rotation_yaw_max = 90°; pitch_range_flag = 1, rotation_pitch_min = -20°, rotation_pitch_max = 80°; roll_range_flag = 1, rotation_roll_min = 20°, rotation_roll_max = 70°.
[0198] S303. Encapsulate the free-viewpoint video and the free-viewpoint video presentation instruction information into an immersive media file.
[0199] For a detailed implementation of step S303, please refer to Figure 1d The implementation methods for encapsulating medium-length video files will not be described in detail here.
[0200] In one implementation, the content production device supports streaming transmission. After obtaining the media file of the immersive media, the content production device slices the media file to obtain multiple media segments; and generates a transmission signaling file for the immersive media, the transmission signaling file containing descriptive information corresponding to the presentation indication information of the free-viewpoint video; wherein the descriptive information includes: a free-viewpoint camera descriptor and a free-viewpoint video angle range descriptor.
[0201] The free-view camera descriptor is used to indicate the parameters of each camera corresponding to the free-view video; the free-view video angle range descriptor is used to indicate the range of view rotation supported by the free-view video; see Tables 11 and 12 above for details, which will not be repeated here.
[0202] The following is a complete example illustrating the data processing method for immersive media provided in this application:
[0203] In one embodiment, the free-view video consists of images captured by Camera1-Camera3.
[0204] Content production equipment: generates free-viewpoint video bitstreams and encapsulates these bitstreams into media files. Depending on the application of the free-viewpoint video, it adds corresponding metadata information (i.e., presentation indication information) to the free-viewpoint video file, which may include:
[0205] Indicates parameter information related to the shooting camera; indicates the range of angle rotation supported by free-view video;
[0206] AvsFreeViewInfoBox:
[0207] duplicated_camera_params=1;
[0208] Camera1, Camera2, Camera3: camera_type==1;
[0209] perspective_horizontal_fov=180; perspective_aspect_ratio=1.
[0210] {Camera1: ID=1; Pos=(100,0,100); orientation=(0,0,0)};
[0211] {Camera2: ID=2; Pos=(100,100,100); orientation=(0.5,0.5,0)};
[0212] {Camera3: ID=3; Pos=(0,0,100); orientation=(0.5,0.5,-0.5)}.
[0213] AvsFreeViewAngleRangeBox:
[0214] yaw_range_flag=1; rotation_yaw_min=-90°; rotation_yaw_max=90°.
[0215] Furthermore, if the content production device supports streaming, the media file is sliced into media segments suitable for streaming (according to the specifications of the transmission standard), and corresponding transmission signaling is generated, which indicates the following information:
[0216] Indicates camera parameter information and the range of angle rotation supported by free-view video;
[0217] Fill in the corresponding fields in the AvsFreeViewCamInfo and AvsFreeViewAngleRange descriptors in the DASH signaling according to the values of the data boxes in the metadata information mentioned above.
[0218] After completing the above steps, the content creation device will transmit the signaling file to the content consumption device.
[0219] Content consumption device: In one implementation, the content consumption device downloads the complete file and plays it locally. During playback, based on the parameters of each camera in the AvsFreeViewInfoBox and the viewer's virtual perspective, it selects the appropriate texture map and depth map corresponding to the camera for decoding and synthesizing the corresponding image. Simultaneously, it limits the viewer's viewing range (e.g., the range of virtual perspective dragging) based on the viewing angle range in the AvsFreeViewAngleRangeBox. Specifically, duplicated_camera_params = 1 indicates that the IntCameraInfoStruct, camera_resolution_x, camera_resolution_y, depth_downsample_factor, texture_padding_size, and depth_padding_size fields are identical for Camera1, Camera2, and Camera3. `camera_type == 1` indicates that the projection algorithm type is perspective projection. `perspective_horizontal_fov = 180` indicates that the horizontal range of the viewport area is (0, 180°) when projecting using perspective. `perspective_aspect_ratio = 1` indicates that the aspect ratio of the viewport is 1 when projecting using perspective. The `Pos` field of each camera indicates the coordinate position of each camera in the free-view video, and the `orientation` field of each camera indicates the orientation of each camera in the free-view video. `yaw_range_flag = 1`, `rotation_yaw_min = -90°`, and `rotation_yaw_max = 90°` indicate that the yaw angle range of the free-view video is [-90°, 90°].
[0220] In another implementation, the content consumption device establishes a streaming connection with the content production device. Based on the parameters of each camera in the AvsFreeViewCamInfo (free-view camera descriptor) and the viewer's virtual viewing angle, the content requests the appropriate video streams containing the texture and depth maps corresponding to the cameras. Upon receiving the corresponding video streams, the content is composited into a single image. Simultaneously, the viewing angle range in the AvsFreeViewAngleRange (free-view video angle range descriptor) is used to limit the viewer's viewing area (e.g., the range of virtual view dragging).
[0221] This application extends the data box and media presentation description file of immersive media. The data processing method for immersive media includes: acquiring image data captured by one or more cameras and encoding the image data into free-viewpoint video; adding presentation indication information to the free-viewpoint video according to its application format; the presentation indication information of the free-viewpoint video includes indication information of the viewing content range of the free-viewpoint video; and encapsulating the free-viewpoint video and its presentation indication information into a media file for immersive media. It is evident that by adding indication information of the viewing content range of the free-viewpoint video to the presentation indication information, the viewing content range of the free-viewpoint video can be indicated, guiding viewers to reasonably adjust their viewing angle within the indicated range to obtain a better viewing effect. Furthermore, the indication method for camera parameters is optimized by combining the free-viewpoint data box and camera frame information structure with the description of the free-viewpoint camera descriptor.
[0222] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.
[0223] Please see Figure 4 , Figure 4 This illustration shows a schematic diagram of the structure of a data processing apparatus for immersive media according to an exemplary embodiment of this application; the data processing apparatus for immersive media can be a computer program (including program code) running on a content consumption device, for example, the data processing apparatus for immersive media can be application software in the content consumption device. Figure 4 As shown, the immersive media data processing device includes an acquisition unit 401 and a processing unit 402.
[0224] Please see Figure 4 In one exemplary embodiment, the various units are described in detail below:
[0225] The acquisition unit 401 is used to acquire the media file of the immersive media, the media file containing presentation indication information of the free-view video, the presentation indication information including the indication information of the viewing content range of the free-view video;
[0226] The processing unit 402 is used to display the free-view video according to the presentation instruction information of the free-view video.
[0227] In one implementation, the indication information for the viewing content range of the free-view video includes the indication information for the viewing angle rotation range supported by the free-view video; the presentation indication information for the free-view video is metadata information, which contains a free-view range data box; the free-view range data box contains the indication information for the viewing angle rotation range supported by the free-view video.
[0228] In one implementation, the free-viewpoint range data box includes a yaw angle flag field and a yaw angle range field;
[0229] The yaw angle flag field indicates the yaw angle of the virtual viewpoint of the viewer while watching immersive media; the yaw angle range field indicates the range of yaw angles to which the virtual viewpoint of the viewer while watching immersive media belongs.
[0230] When the yaw angle flag field is set to the first setting value, it means that the yaw angle of the virtual viewpoint of the immersive media viewer is limited to the yaw angle range indicated by the yaw angle range field; when the yaw angle flag field is set to the second setting value, it means that the yaw angle of the virtual viewpoint of the immersive media viewer is not limited.
[0231] In one implementation, the free-viewpoint range data box includes a pitch angle flag field and a pitch angle range field;
[0232] The pitch angle flag field indicates the pitch angle of the virtual viewpoint of the viewer when watching immersive media; the pitch angle range field indicates the range of pitch angles to which the virtual viewpoint of the viewer belongs when watching immersive media.
[0233] When the pitch angle flag field is set to the first setting value, it means that the virtual viewing angle of the immersive media viewer is limited to the pitch angle range indicated by the pitch angle range field; when the pitch angle flag field is set to the second setting value, it means that the virtual viewing angle of the immersive media viewer is not limited.
[0234] In one implementation, the free-viewpoint range data box includes a roll angle flag field and a roll angle range field;
[0235] The roll angle flag field indicates the roll angle of the virtual viewpoint of the viewer while watching immersive media; the roll angle range field indicates the roll angle range of the virtual viewpoint of the viewer while watching immersive media.
[0236] When the roll angle flag field is set to the first setting value, it means that the virtual view roll angle of the immersive media viewer is limited to the roll angle range indicated by the roll angle range field; when the roll angle flag field is set to the second setting value, it means that the virtual view roll angle of the immersive media viewer is not limited.
[0237] In one implementation, the free-viewpoint range data box includes a depth range flag field and a depth range field;
[0238] The Depth Range Flag field is used to indicate the depth of the virtual viewpoint of the viewer when watching immersive media; the Depth Range field is used to indicate the depth range to which the virtual viewpoint of the viewer belongs when watching immersive media.
[0239] When the depth range flag field is set to the first setting value, it means that the virtual viewing depth of the immersive media viewer is limited by the depth range indicated by the depth range field; when the depth range flag field is set to the second setting value, it means that the virtual viewing depth of the immersive media viewer is not limited.
[0240] In one implementation, the free-viewpoint range data box includes a three-dimensional rotation quaternion; the three-dimensional rotation quaternion is used to indicate the three-dimensional rotation range of the virtual viewpoint of the viewer of the immersive media while viewing it.
[0241] In one implementation, the three-dimensional rotation quaternion includes an x-component, a y-component, and a z-component, and the free view range data box contains a range field for the x-component of the quaternion, a range field for the y-component of the quaternion, and a range field for the z-component of the quaternion.
[0242] The range field for the x-component of a quaternion is used to indicate the range of values for the x-component in a three-dimensional rotated quaternion.
[0243] The range field for the y-component of a quaternion is used to indicate the range of values for the y-component in a three-dimensional rotated quaternion.
[0244] The range field for the z-component of a quaternion is used to indicate the range of values for the z-component in a three-dimensional rotated quaternion.
[0245] In one implementation, the free-viewpoint range data box includes a vector origin field, a vector endpoint field, and a yaw angle range field;
[0246] The vector origin and vector endpoint fields are used to determine the starting vector, while the starting vector and yaw angle range fields are used to indicate the range of virtual viewpoints for viewers of immersive media.
[0247] In one implementation, the metadata information also includes a three-dimensional boundary structure, which is used to indicate the boundary range of the scene to which the virtual viewpoint of the viewer of the immersive media belongs when viewing, and the boundary range includes a height range; the free viewpoint range data box also includes a height range flag field and a height range field;
[0248] The height range flag field is used to indicate the virtual viewing height of the viewer when watching immersive media; the height range field is used to indicate the height range of the virtual viewing height of the viewer when watching immersive media.
[0249] When the height range flag field is set to the first setting value, it means that the virtual viewing angle of the immersive media viewer is limited in height by the height range indicated by the height range field; when the height range flag field is set to the second setting value, it means that the virtual viewing angle of the immersive media viewer is not limited in height within the height range indicated by the three-dimensional boundary structure.
[0250] In one implementation, the free-viewpoint video is encapsulated in at least one track, the at least one track containing a target track, the target track containing image data of the free-viewpoint video;
[0251] When the target track contains a free-viewing-range data box, it means that the viewing angle corresponding to the image data in the target track is limited; when the target track does not contain a free-viewing-range data box, it means that the viewing angle corresponding to the image data in the target track is not limited.
[0252] In one implementation, the metadata information includes a free-viewpoint data box, and a free-viewpoint range data box is contained within the free-viewpoint data box; the free-viewpoint data box also includes an angle range flag field;
[0253] When the angle range flag field is set to the first set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint data box is limited, and the limited range of the viewing angle is indicated by the free-viewpoint data box; when the angle range flag field is set to the second set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint data box is not limited.
[0254] In one implementation, the metadata information includes a free-viewpoint boundary data box, and a free-viewpoint range data box is contained within the free-viewpoint boundary data box; the free-viewpoint boundary data box also includes an angle range flag field.
[0255] When the angle range flag field is set to the first set value, it indicates that the viewing angle of the image data corresponding to the free viewpoint boundary data box is limited, and the limited range of the viewing angle is indicated by the free viewpoint range data box; when the angle range flag field is set to the second set value, it indicates that the viewing angle of the image data corresponding to the free viewpoint boundary data box is not limited.
[0256] In one embodiment, the processing unit 402 is configured to display the free-viewpoint video according to the presentation instruction information of the free-viewpoint video, specifically configured to:
[0257] During the display of image data, the rotation range of the virtual viewpoint for viewers of immersive media is limited according to the viewpoint rotation range indication information supported by free-viewpoint video.
[0258] In one implementation, the indication information for the viewing content range of the free-viewpoint video includes indication information for the viewing content of the free-viewpoint video; the viewing content of the free-viewpoint video is generated based on image data captured by one or more cameras; the presentation indication information of the free-viewpoint video is metadata information, which contains a free-viewpoint data box; the free-viewpoint data box contains indication information for the viewing content of the free-viewpoint video.
[0259] In one implementation, the free-view data box includes camera parameter indication information, which is used to indicate the viewing content of the free-view video; the camera parameter indication information includes camera parameters and repeating camera parameter indication fields.
[0260] When the repeat camera parameter indicator field is set to the first setting value, it means that the camera parameters in the free-view data box are effective for all cameras; when the repeat camera parameter indicator field is set to the second setting value, it means that different cameras in the free-view data box have their own camera parameters.
[0261] In one embodiment, the processing unit 402 is configured to display the free-viewpoint video according to the presentation instruction information of the free-viewpoint video, specifically configured to:
[0262] Based on the virtual perspective of the viewer while watching immersive media, determine the image data required for viewing from the media file;
[0263] The image data is decoded and displayed according to the camera parameter instructions.
[0264] In one implementation, the immersive media file is sliced into multiple media segments; the acquisition unit 401 is further configured to:
[0265] Obtain the transmission signaling file of the immersive media, which contains descriptive information corresponding to the presentation instructions for the free-viewpoint video.
[0266] In one implementation, the transmission signaling file includes at least one of the following: a free-view camera descriptor and a free-view video angle range descriptor;
[0267] Among them, the free-view camera descriptor is used to indicate camera parameters; the free-view video angle range descriptor is used to indicate the boundary information of the free-view scene in the free-view video.
[0268] In one implementation, the transmission signaling file includes a representation layer and an adaptive set layer, wherein the adaptive set layer includes one or more representation layers; when description information exists in the target adaptive set layer, it is used to describe all representation layers in the target adaptive set layer; when description information exists in the target representation layer, it is used to describe the target representation layer.
[0269] In one embodiment, the acquisition unit 401 is used to acquire the media file of the immersive media, specifically for:
[0270] Based on the virtual perspective of the viewer of immersive media and the description information in the transmission signaling file, determine the media segment corresponding to the image data required for viewing;
[0271] The determined media segments are retrieved using streaming.
[0272] According to one embodiment of this application, Figure 2 The data processing method for immersive media shown can be partially implemented by... Figure 4 The data processing is performed by individual units within the immersive media data processing device shown. For example, Figure 2 Step S201 shown can be performed by Figure 4 The acquisition unit 401 shown is executed, and step S202 can be performed by... Figure 4 The processing unit 402 shown executes. Figure 4 The various units in the immersive media data processing apparatus shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the immersive media data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.
[0273] According to another embodiment of this application, the following can be executed by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 2 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 4 The data processing apparatus for immersive media shown herein, and the data processing method for immersive media for implementing embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.
[0274] Based on the same inventive concept, the data processing device for immersive media provided in the embodiments of this application solves the problem in a similar principle and with similar beneficial effects as the data processing method for immersive media in the embodiments of this application. For details, please refer to the implementation principle and beneficial effects of the method. For the sake of brevity, these will not be repeated here.
[0275] Please see Figure 5 , Figure 5 This illustration shows a schematic diagram of another immersive media data processing apparatus provided in an exemplary embodiment of this application; the immersive media data processing apparatus can be a computer program (including program code) running on a content production device, for example, the immersive media data processing apparatus can be application software in the content production device. Figure 5 As shown, the immersive media data processing apparatus includes an acquisition unit 501 and a processing unit 502. (See also...) Figure 5 The detailed descriptions of each unit are as follows:
[0276] The acquisition unit 501 is used to acquire image data captured by one or more cameras and encode the image data into free-viewpoint video.
[0277] The processing unit 502 is configured to add presentation instruction information to the free-view video according to the application format of the free-view video; the presentation instruction information of the free-view video includes instruction information on the viewing content range of the free-view video; and to encapsulate the free-view video and the presentation instruction information of the free-view video into a media file for immersive media.
[0278] In one embodiment, the processing unit 502 is further configured to:
[0279] Slicing a media file to obtain multiple media segments; and,
[0280] Generate a transmission signaling file for immersive media, which contains descriptive information corresponding to the presentation instructions for free-viewpoint video.
[0281] According to one embodiment of this application, Figure 3 The data processing method for immersive media shown can be partially implemented by... Figure 5 The data processing is performed by individual units within the immersive media data processing device shown. For example, Figure 3 Step S301 shown can be performed by Figure 5 The acquisition unit 501 shown is executed, and steps S302 and S303 can be performed by... Figure 5 The processing unit 502 shown is executed. Figure 5 The various units in the immersive media data processing apparatus shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the immersive media data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.
[0282] According to another embodiment of this application, the following can be executed by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 3 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 5 The data processing apparatus for immersive media shown herein, and the data processing method for immersive media for implementing embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.
[0283] Based on the same inventive concept, the data processing device for immersive media provided in the embodiments of this application solves the problem in a similar principle and with similar beneficial effects as the data processing method for immersive media in the embodiments of this application. For details, please refer to the implementation principle and beneficial effects of the method. For the sake of brevity, these will not be repeated here.
[0284] Figure 6 This illustration shows a schematic diagram of the structure of a content consumption device provided in an exemplary embodiment of this application; the content consumption device can refer to a computer device used by a user of immersive media, and the computer device can be a terminal (such as a PC, a smart mobile device (such as a smartphone), a VR device (such as a VR headset, VR glasses, etc.)). Figure 6 As shown, the content consumption device includes a receiver 601, a processor 602, a memory 603, and a display / playback device 604. Wherein:
[0285] Receiver 601 is used to enable decoding and transmission interaction with other devices, specifically for the transmission of immersive media between content creation devices and content consumption devices. That is, the content consumption device receives the relevant media resources for immersive media transmitted by the content creation device through receiver 601.
[0286] Processor 602 (or CPU (Central Processing Unit)) is the processing core of the content production device. Processor 602 is adapted to implement one or more program instructions, specifically to load and execute one or more program instructions to achieve... Figure 2 The flowchart illustrates the data processing method for immersive media.
[0287] Memory 603 is a memory device in the content consumption device used to store programs and media resources. It is understood that memory 603 here can include the built-in storage medium of the content consumption device, or it can include extended storage media supported by the content consumption device. It should be noted that memory 603 can be high-speed RAM, or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one memory located remotely from the aforementioned processor. Memory 603 provides storage space for storing the operating system of the content consumption device. Furthermore, this storage space is also used to store computer programs, which include program instructions adapted to be called and executed by the processor to perform the various steps of the immersive media data processing method. In addition, memory 603 can also be used to store the three-dimensional image of the immersive media formed after processor processing, the audio content corresponding to the three-dimensional image, and information required for rendering the three-dimensional image and audio content.
[0288] Display / playback device 604 is used to output rendered sound and 3D images.
[0289] Please see again Figure 6 The processor 602 may include a parser 621, a decoder 622, a converter 623, and a renderer 624; wherein:
[0290] The parser 621 is used to depackage and encapsulate the rendering media from the content production device. Specifically, it depackages the media file resources according to the file format requirements of immersive media to obtain audio and video streams, and provides the audio and video streams to the decoder 622.
[0291] Decoder 622 decodes the audio stream to obtain audio content, which is then provided to the renderer for audio rendering. Additionally, decoder 622 decodes the video stream to obtain a 2D image. Based on the metadata provided by the media presentation description information, if the metadata indicates that the immersive media has undergone a region encapsulation process, the 2D image refers to an encapsulated image; if the metadata indicates that the immersive media has not undergone a region encapsulation process, the planar image refers to a projected image.
[0292] Converter 623 is used to convert 2D images into 3D images. If the immersive media has undergone a region encapsulation process, converter 623 will first decapsulate the encapsulated image to obtain a projected image. Then, the projected image will be reconstructed to obtain a 3D image. If the rendering media has not undergone a region encapsulation process, converter 623 will directly reconstruct the projected image to obtain a 3D image.
[0293] Renderer 624 is used to render the audio content and 3D images of immersive media. Specifically, it renders the audio content and 3D images based on the metadata related to rendering and viewing in the media presentation description information, and then outputs the rendered content to the display / playback device.
[0294] In one exemplary embodiment, the processor 602 (specifically, the devices included in the processor) executes instructions by calling one or more instructions stored in memory. Figure 2 The illustrated steps of the immersive media data processing method. Specifically, the memory stores one or more first instructions, which are adapted to be loaded by the processor 602 and executed in the following steps:
[0295] Acquire the media file of the immersive media, which contains presentation instructions for the free-viewpoint video, including instructions on the viewing range of the free-viewpoint video.
[0296] Display the free-view video according to the presentation instructions.
[0297] In one embodiment, the indication information for the viewing content range of the free-view video includes the indication information for the viewing rotation range supported by the free-view video; the presentation indication information for the free-view video is metadata information, which contains a free-view range data box; the free-view range data box contains the indication information for the viewing rotation range supported by the free-view video.
[0298] In one implementation, the free-viewpoint range data box includes a yaw angle flag field and a yaw angle range field;
[0299] The yaw angle flag field indicates the yaw angle of the virtual viewpoint of the viewer while watching immersive media; the yaw angle range field indicates the range of yaw angles to which the virtual viewpoint of the viewer while watching immersive media belongs.
[0300] When the yaw angle flag field is set to the first setting value, it means that the yaw angle of the virtual viewpoint of the immersive media viewer is limited to the yaw angle range indicated by the yaw angle range field; when the yaw angle flag field is set to the second setting value, it means that the yaw angle of the virtual viewpoint of the immersive media viewer is not limited.
[0301] In one implementation, the free-viewpoint range data box includes a pitch angle flag field and a pitch angle range field;
[0302] The pitch angle flag field indicates the pitch angle of the virtual viewpoint of the viewer when watching immersive media; the pitch angle range field indicates the range of pitch angles to which the virtual viewpoint of the viewer belongs when watching immersive media.
[0303] When the pitch angle flag field is set to the first setting value, it means that the virtual viewing angle of the immersive media viewer is limited to the pitch angle range indicated by the pitch angle range field; when the pitch angle flag field is set to the second setting value, it means that the virtual viewing angle of the immersive media viewer is not limited.
[0304] In one implementation, the free-viewpoint range data box includes a roll angle flag field and a roll angle range field;
[0305] The roll angle flag field indicates the roll angle of the virtual viewpoint of the viewer while watching immersive media; the roll angle range field indicates the roll angle range of the virtual viewpoint of the viewer while watching immersive media.
[0306] When the roll angle flag field is set to the first setting value, it means that the virtual view roll angle of the immersive media viewer is limited to the roll angle range indicated by the roll angle range field; when the roll angle flag field is set to the second setting value, it means that the virtual view roll angle of the immersive media viewer is not limited.
[0307] In one implementation, the free-viewpoint range data box includes a depth range flag field and a depth range field;
[0308] The Depth Range Flag field is used to indicate the depth of the virtual viewpoint of the viewer when watching immersive media; the Depth Range field is used to indicate the depth range to which the virtual viewpoint of the viewer belongs when watching immersive media.
[0309] When the depth range flag field is set to the first setting value, it means that the virtual viewing depth of the immersive media viewer is limited by the depth range indicated by the depth range field; when the depth range flag field is set to the second setting value, it means that the virtual viewing depth of the immersive media viewer is not limited.
[0310] In one implementation, the free-viewpoint range data box includes a three-dimensional rotation quaternion; the three-dimensional rotation quaternion is used to indicate the three-dimensional rotation range of the virtual viewpoint of the viewer of the immersive media while viewing.
[0311] In one embodiment, the three-dimensional rotation quaternion includes an x-component, a y-component, and a z-component, and the free view range data box contains a range field for the x-component of the quaternion, a range field for the y-component of the quaternion, and a range field for the z-component of the quaternion.
[0312] The range field for the x-component of a quaternion is used to indicate the range of values for the x-component in a three-dimensional rotated quaternion.
[0313] The range field for the y-component of a quaternion is used to indicate the range of values for the y-component in a three-dimensional rotated quaternion.
[0314] The range field for the z-component of a quaternion is used to indicate the range of values for the z-component in a three-dimensional rotated quaternion.
[0315] In one implementation, the free-viewpoint range data box includes a vector origin field, a vector endpoint field, and a yaw angle range field;
[0316] The vector origin and vector endpoint fields are used to determine the starting vector, while the starting vector and yaw angle range fields are used to indicate the range of virtual viewpoints for viewers of immersive media.
[0317] In one embodiment, the metadata information further includes a three-dimensional boundary structure, which is used to indicate the boundary range of the scene to which the virtual viewpoint of the viewer of the immersive media belongs when watching. The boundary range includes a height range. The free viewpoint range data box also includes a height range flag field and a height range field.
[0318] The height range flag field is used to indicate the virtual viewing height of the viewer when watching immersive media; the height range field is used to indicate the height range of the virtual viewing height of the viewer when watching immersive media.
[0319] When the height range flag field is set to the first setting value, it means that the virtual viewing angle of the immersive media viewer is limited in height by the height range indicated by the height range field; when the height range flag field is set to the second setting value, it means that the virtual viewing angle of the immersive media viewer is not limited in height within the height range indicated by the three-dimensional boundary structure.
[0320] In one implementation, the free-viewpoint video is encapsulated in at least one track, the at least one track containing a target track, and the target track containing image data of the free-viewpoint video;
[0321] When the target track contains a free-viewing-range data box, it means that the viewing angle corresponding to the image data in the target track is limited; when the target track does not contain a free-viewing-range data box, it means that the viewing angle corresponding to the image data in the target track is not limited.
[0322] In one implementation, the metadata information includes a free-viewpoint data box, and a free-viewpoint range data box is contained within the free-viewpoint data box; the free-viewpoint data box also includes an angle range flag field;
[0323] When the angle range flag field is set to the first set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint data box is limited, and the limited range of the viewing angle is indicated by the free-viewpoint data box; when the angle range flag field is set to the second set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint data box is not limited.
[0324] In one implementation, the metadata information includes a free-viewpoint boundary data box, and a free-viewpoint range data box is contained within the free-viewpoint boundary data box; the free-viewpoint boundary data box also includes an angle range flag field.
[0325] When the angle range flag field is set to the first set value, it indicates that the viewing angle of the image data corresponding to the free viewpoint boundary data box is limited, and the limited range of the viewing angle is indicated by the free viewpoint range data box; when the angle range flag field is set to the second set value, it indicates that the viewing angle of the image data corresponding to the free viewpoint boundary data box is not limited.
[0326] In one embodiment, the processor 602 displays the free-viewpoint video according to the presentation instruction information of the free-viewpoint video. The specific implementation method is as follows:
[0327] During the display of image data, the rotation range of the virtual viewpoint for viewers of immersive media is limited according to the viewpoint rotation range indication information supported by free-viewpoint video.
[0328] In one embodiment, the indication information for the viewing content range of the free-viewpoint video includes indication information for the viewing content of the free-viewpoint video; the viewing content of the free-viewpoint video is generated based on image data captured by one or more cameras; the presentation indication information of the free-viewpoint video is metadata information, which contains a free-viewpoint data box; the free-viewpoint data box contains indication information for the viewing content of the free-viewpoint video.
[0329] In one embodiment, the free-view data box contains camera parameter indication information, which is used to indicate the viewing content of the free-view video; the camera parameter indication information includes camera parameters and a repeating camera parameter indication field.
[0330] When the repeat camera parameter indicator field is set to the first setting value, it means that the camera parameters in the free-view data box are effective for all cameras; when the repeat camera parameter indicator field is set to the second setting value, it means that different cameras in the free-view data box have their own camera parameters.
[0331] In one embodiment, the processor 602 displays the free-viewpoint video according to the presentation instruction information of the free-viewpoint video. The specific implementation method is as follows:
[0332] Based on the virtual perspective of the viewer while watching immersive media, determine the image data required for viewing from the media file;
[0333] The image data is decoded and displayed according to the camera parameter instructions.
[0334] In one embodiment, the immersive media file is sliced into multiple media segments; the computer program in memory 603 is loaded by processor 602 and also performs the following steps:
[0335] Obtain the transmission signaling file of the immersive media, which contains descriptive information corresponding to the presentation instructions for the free-viewpoint video.
[0336] In one implementation, the transmission signaling file includes at least one of the following: a free-view camera descriptor and a free-view video angle range descriptor;
[0337] Among them, the free-view camera descriptor is used to indicate camera parameters; the free-view video angle range descriptor is used to indicate the boundary information of the free-view scene in the free-view video.
[0338] In one embodiment, the transmission signaling file includes a representation layer and an adaptive set layer, wherein the adaptive set layer includes one or more representation layers; when description information exists in the target adaptive set layer, it is used to describe all representation layers in the target adaptive set layer; when description information exists in the target representation layer, it is used to describe the target representation layer.
[0339] In one embodiment, the processor 602 acquires the media file of the immersive media as follows:
[0340] Based on the virtual perspective of the viewer of immersive media and the description information in the transmission signaling file, determine the media segment corresponding to the image data required for viewing;
[0341] The determined media segments are retrieved using streaming.
[0342] Based on the same inventive concept, the principle and beneficial effects of the content consumption device provided in the embodiments of this application are similar to the principle and beneficial effects of the immersive media processing method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, they will not be repeated here.
[0343] Figure 7 This illustration shows a schematic diagram of a content creation device provided in an exemplary embodiment of this application; the content creation device may refer to a computer device used by an immersive media provider, which may be a terminal (such as a PC, a smart mobile device (such as a smartphone) or a server. Figure 7 As shown, the content creation device includes a capture device 701, a processor 702, a memory 703, and a transmitter 704. Wherein:
[0344] The capture device 701 is used to acquire raw data (including audio and video content synchronized in time and space) of real-world sound-visual scenes to obtain immersive media. The capture device 701 may include, but is not limited to, audio devices, camera devices, and sensing devices. Audio devices may include audio sensors, microphones, etc. Camera devices may include ordinary cameras, stereo cameras, light field cameras, etc. Sensing devices may include laser devices, radar devices, etc.
[0345] Processor 702 (or CPU (Central Processing Unit)) is the processing core of the content production device. Processor 702 is adapted to implement one or more program instructions, specifically to load and execute one or more program instructions to achieve... Figure 3 The flowchart illustrates the data processing method for immersive media.
[0346] Memory 703 is a memory device in the content creation apparatus used to store programs and media resources. It is understood that memory 703 here can include both the built-in storage medium of the content creation apparatus and extended storage media supported by the content creation apparatus. It should be noted that the memory can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one memory located remotely from the aforementioned processor. The memory provides storage space for storing the operating system of the content creation apparatus. Furthermore, this storage space is also used to store computer programs, which include program instructions adapted to be invoked and executed by the processor to perform the various steps of the immersive media data processing method. Additionally, memory 703 can also be used to store immersive media files formed after processing by the processor, which include media file resources and media presentation description information.
[0347] The transmitter 704 is used to enable transmission and interaction between the content creation device and other devices, specifically to facilitate the transmission of immersive media between the content creation device and the content playback device. That is, the content creation device uses the transmitter 704 to transmit relevant media resources for immersive media to the content playback device.
[0348] Please see again Figure 7 The processor 702 may include a converter 721, an encoder 722, and a packager 723; wherein:
[0349] Converter 721 performs a series of conversion processes on captured video content to make it suitable for immersive media video encoding. The conversion processes may include stitching and projection; optionally, they may also include region encapsulation. Converter 721 can convert captured 3D video content into 2D images and provide them to the encoder for video encoding.
[0350] Encoder 722 is used to encode the captured audio content to form an audio stream for immersive media. It is also used to encode the 2D image obtained by converter 721 to obtain a video stream.
[0351] The encapsulator 723 encapsulates audio and video streams into a file container according to the immersive media file format (such as ISOBMFF) to form an immersive media file resource. This media file resource can be a media file or a media segment forming an immersive media file. It also records the metadata of the immersive media file resource using media presentation description information according to the immersive media file format requirements. The encapsulated immersive media file obtained by the encapsulator is stored in memory and provided to the content playback device as needed for immersive media presentation.
[0352] The processor 702 (specifically, the various components within the processor) executes instructions by calling one or more instructions from memory. Figure 4 The illustrated steps of the immersive media data processing method are as follows. Specifically, memory 703 stores one or more first instructions, which are adapted to be loaded by processor 702 and executed in the following steps:
[0353] Acquire image data from one or more cameras and encode the image data into free-viewpoint video;
[0354] Add presentation indication information to free-viewpoint videos based on their application format; the presentation indication information for free-viewpoint videos includes indications of the viewing content range of the free-viewpoint video.
[0355] Encapsulate free-viewpoint video and free-viewpoint video presentation instructions into immersive media files.
[0356] In one embodiment, the computer program in memory 703 is loaded by processor 702 and further performs the following steps:
[0357] Slicing a media file to obtain multiple media segments; and,
[0358] Generate a transmission signaling file for immersive media, which contains descriptive information corresponding to the presentation instructions for free-viewpoint video.
[0359] Based on the same inventive concept, the principle and beneficial effects of the content production device provided in the embodiments of this application are similar to the principle and beneficial effects of the immersive media processing method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, these will not be repeated here.
[0360] This application also provides a computer-readable storage medium storing one or more instructions adapted for a processor to load and execute the immersive media data processing method of the above-described method embodiments.
[0361] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the immersive media data processing method described in the above method embodiments.
[0362] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned immersive media data processing method.
[0363] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.
[0364] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.
[0365] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.
[0366] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application.
Claims
1. A data processing method for immersive media, characterized in that, include Obtain immersive media files, the media files containing presentation indication information for free-viewpoint video, the presentation indication information including indication information for the viewing content range of the free-viewpoint video, the indication information for the viewing content range including camera parameter indication information existing in the free-viewpoint data box, the camera parameter indication information being used to indicate the viewing content of the free-viewpoint video, the viewing content being generated based on image data captured by one or more cameras; the camera parameter indication information including camera parameters and repeating camera parameter indication fields; When the repeat camera parameter indicator field takes the first set value, it means that the camera parameters in the free-view data box are effective for all cameras; when the repeat camera parameter indicator field takes the second set value, it means that different cameras in the free-view data box have their own camera parameters. The free-view video is displayed according to the presentation instructions.
2. The method as described in claim 1, characterized in that, The indication information for the viewing content range of the free-view video includes the indication information for the viewing angle rotation range supported by the free-view video; the presentation indication information of the free-view video is metadata information, which contains a free-view range data box; the free-view range data box contains the indication information for the viewing angle rotation range supported by the free-view video.
3. The method as described in claim 2, characterized in that, The free-viewpoint range data box includes a yaw angle flag field and a yaw angle range field; The yaw angle flag field is used to indicate the yaw angle of the virtual viewpoint of the viewer of the immersive media; the yaw angle range field is used to indicate the yaw angle range to which the virtual viewpoint of the viewer of the immersive media belongs. When the yaw angle flag field is set to a first set value, it means that the yaw angle of the virtual viewpoint of the viewer of the immersive media is limited to the yaw angle range indicated by the yaw angle range field; when the yaw angle flag field is set to a second set value, it means that the yaw angle of the virtual viewpoint of the viewer of the immersive media is not limited.
4. The method as described in claim 2, characterized in that, The free-viewpoint range data box includes a pitch angle flag field and a pitch angle range field; The pitch angle flag field is used to indicate the pitch angle of the virtual viewpoint of the viewer of the immersive media; the pitch angle range field is used to indicate the pitch angle range to which the virtual viewpoint of the viewer of the immersive media belongs. When the pitch angle flag field is set to a first set value, it means that the virtual viewing angle of the viewer of the immersive media is limited to the pitch angle range indicated by the pitch angle range field; when the pitch angle flag field is set to a second set value, it means that the virtual viewing angle of the viewer of the immersive media is not limited.
5. The method as described in claim 2, characterized in that, The free-viewpoint range data box includes a roll angle flag field and a roll angle range field; The roll angle flag field is used to indicate the roll angle of the virtual viewpoint of the viewer of the immersive media while watching; the roll angle range field is used to indicate the roll angle range to which the virtual viewpoint of the viewer of the immersive media belongs while watching. When the roll angle flag field is set to a first set value, it means that the virtual view roll angle of the viewer of the immersive media is limited to the roll angle range indicated by the roll angle range field; when the roll angle flag field is set to a second set value, it means that the virtual view roll angle of the viewer of the immersive media is not limited.
6. The method as described in claim 2, characterized in that, The free-viewpoint range data box includes a depth range flag field and a depth range field; The depth range flag field is used to indicate the virtual perspective depth of the viewer of the immersive media; the depth range field is used to indicate the depth range to which the virtual perspective of the viewer of the immersive media belongs. When the depth range flag field is set to a first set value, it means that the virtual viewing angle depth of the viewer of the immersive media is limited by the depth range indicated by the depth range field; when the depth range flag field is set to a second set value, it means that the virtual viewing angle depth of the viewer of the immersive media is not limited.
7. The method as described in claim 2, characterized in that, The free-viewpoint range data box contains a three-dimensional rotation quaternion; the three-dimensional rotation quaternion is used to indicate the three-dimensional rotation range of the virtual viewpoint of the viewer of the immersive media.
8. The method as described in claim 7, characterized in that, The three-dimensional rotation quaternion includes x-component, y-component, and z-component, and the free view range data box contains a range field for the x-component of the quaternion, a range field for the y-component of the quaternion, and a range field for the z-component of the quaternion. The range field of the quaternion x component is used to indicate the range of values for the x component in the three-dimensional rotated quaternion; The range field of the quaternion y-component is used to indicate the range of values for the y-component in the three-dimensional rotated quaternion; The range field of the quaternion z-component is used to indicate the range of values for the z-component in the three-dimensional rotated quaternion.
9. The method as described in claim 2, characterized in that, The free-viewpoint range data box includes a vector origin field, a vector endpoint field, and a yaw angle range field; The vector origin field and the vector endpoint field are used to determine the starting vector, and the starting vector and the yaw angle range field are used to indicate the range of virtual viewpoints for the viewer of the immersive media.
10. The method as described in claim 9, characterized in that, The metadata information also includes a three-dimensional boundary structure, which is used to indicate the boundary range of the scene to which the virtual perspective of the viewer of the immersive media belongs when watching. The boundary range includes a height range. The free view range data box also includes a height range flag field and a height range field. The height range flag field is used to indicate the virtual viewing angle height of the viewer of the immersive media; the height range field is used to indicate the height range of the virtual viewing angle of the viewer of the immersive media. When the height range flag field is set to a first set value, it means that the virtual viewing angle of the viewer of the immersive media is limited in height by the height range indicated by the height range field; when the height range flag field is set to a second set value, it means that the virtual viewing angle of the viewer of the immersive media is not limited in height within the height range indicated by the three-dimensional boundary structure.
11. The method as described in claim 2, characterized in that, The free-viewpoint video is encapsulated in at least one track, the at least one track containing a target track, and the target track containing the image data of the free-viewpoint video; When the target track contains the free-viewing-angle data box, it indicates that the viewing angle corresponding to the image data in the target track is limited; when the target track does not contain the free-viewing-angle data box, it indicates that the viewing angle corresponding to the image data in the target track is not limited.
12. The method as described in claim 2, characterized in that, The metadata information includes a free-viewpoint data box, and the free-viewpoint range data box is contained within the free-viewpoint data box; the free-viewpoint data box also includes an angle range flag field; When the angle range flag field is set to a first set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint data box is limited, and the free-viewpoint data box indicates the limited range of the viewing angle; when the angle range flag field is set to a second set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint data box is not limited.
13. The method as described in claim 2, characterized in that, The metadata information includes a free-viewpoint boundary data box, and the free-viewpoint range data box is contained within the free-viewpoint boundary data box; the free-viewpoint boundary data box also includes an angle range flag field; When the angle range flag field is set to a first set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint boundary data box is limited, and the limited range of the viewing angle is indicated by the free-viewpoint range data box; when the angle range flag field is set to a second set value, it indicates that the viewing angle of the image data corresponding to the free-viewpoint boundary data box is not limited.
14. The method as described in claim 2, characterized in that, Displaying the free-view video according to the presentation instruction information of the free-view video includes: During the display of the image data, the rotation range of the virtual viewpoint of the viewer of the immersive media is limited according to the viewpoint rotation range indication information supported by the free-viewpoint video.
15. The method as described in claim 1, characterized in that, The indication information for the viewing content range of the free-view video includes indication information for the viewing content of the free-view video; the presentation indication information of the free-view video is metadata information, which includes the free-view data box; the free-view data box includes indication information for the viewing content of the free-view video.
16. The method as described in claim 1, characterized in that, Displaying the free-view video according to the presentation instruction information of the free-view video includes: Based on the virtual perspective of the viewer of the immersive media, determine the image data required for viewing from the media file; The image data is decoded and displayed according to the camera parameter indication information.
17. The method according to any one of claims 1-16, characterized in that, The immersive media file is sliced into multiple media segments; the method further includes: Obtain the transmission signaling file of the immersive media, wherein the transmission signaling file contains descriptive information corresponding to the presentation indication information of the free-viewpoint video.
18. The method as described in claim 17, characterized in that, The transmission signaling file includes at least one of the following: a free-view camera descriptor and a free-view video angle range descriptor; The free-view camera descriptor is used to indicate camera parameters; the free-view video angle range descriptor is used to indicate the boundary information of the free-view scene in the free-view video.
19. The method as described in claim 17, characterized in that, The transmission signaling file includes a representation layer and an adaptive set layer, wherein the adaptive set layer includes one or more representation layers; when the description information exists in the target adaptive set layer, it is used to describe all representation layers in the target adaptive set layer; when the description information exists in the target representation layer, it is used to describe the target representation layer.
20. The method as described in claim 17, characterized in that, The media files for obtaining immersive media include: Based on the virtual perspective of the viewer of the immersive media and the description information in the transmission signaling file, the media segment corresponding to the image data required for viewing is determined. The determined media segments are retrieved using streaming.
21. A data processing method for immersive media, characterized in that, include: Acquire image data from one or more cameras and encode the image data into free-viewpoint video; According to the application format of the free-view video, presentation indication information is added to the free-view video; the presentation indication information of the free-view video includes indication information of the viewing content range of the free-view video, the indication information of the viewing content range includes camera parameter indication information existing in the free-view data box, the camera parameter indication information is used to indicate the viewing content of the free-view video, the viewing content is generated based on image data captured by one or more cameras; the camera parameter indication information includes camera parameters and repeating camera parameter indication fields; When the repeat camera parameter indicator field takes the first set value, it means that the camera parameters in the free-view data box are effective for all cameras; when the repeat camera parameter indicator field takes the second set value, it means that different cameras in the free-view data box have their own camera parameters. The free-view video and its presentation instructions are encapsulated into a media file for immersive media.
22. The method as described in claim 21, characterized in that, The method further includes: The media file is sliced to obtain multiple media segments; and, Generate a transmission signaling file for the immersive media, the transmission signaling file containing descriptive information corresponding to the presentation indication information of the free-viewpoint video.
23. A data processing device for immersive media, characterized in that, The immersive media data processing device includes: An acquisition unit is used to acquire a media file of immersive media, wherein the media file contains presentation indication information for free-viewpoint video, the presentation indication information including indication information of the viewing content range of the free-viewpoint video, the indication information of the viewing content range including camera parameter indication information existing in the free-viewpoint data box, the camera parameter indication information being used to indicate the viewing content of the free-viewpoint video, the viewing content being generated based on image data acquired by one or more cameras; the camera parameter indication information including camera parameters and repeating camera parameter indication fields; When the repeat camera parameter indicator field takes the first set value, it means that the camera parameters in the free-view data box are effective for all cameras; when the repeat camera parameter indicator field takes the second set value, it means that different cameras in the free-view data box have their own camera parameters. The processing unit is used to display the free-view video according to the presentation instruction information of the free-view video.
24. A data processing device for immersive media, characterized in that, The immersive media data processing device includes: An acquisition unit is used to acquire image data captured by one or more cameras and encode the image data into free-viewpoint video. The processing unit is configured to add presentation indication information to the free-viewpoint video according to the application format of the free-viewpoint video; the presentation indication information of the free-viewpoint video includes indication information of the viewing content range of the free-viewpoint video, the indication information of the viewing content range includes camera parameter indication information existing in the free-viewpoint data box, the camera parameter indication information is used to indicate the viewing content of the free-viewpoint video, the viewing content is generated based on image data captured by one or more cameras; the camera parameter indication information includes camera parameters and a repeating camera parameter indication field; when the repeating camera parameter indication field is set to a first preset value, it indicates that the camera parameters in the free-viewpoint data box are effective for all cameras; when the repeating camera parameter indication field is set to a second preset value, it indicates that different cameras in the free-viewpoint data box have their own camera parameters; and to encapsulate the free-viewpoint video and the presentation indication information of the free-viewpoint video into a media file of immersive media.
25. A computer device, characterized in that, include: Storage devices and processors; A memory, wherein a computer program is stored; A processor is configured to load the computer program to implement the data processing method for immersive media as described in any one of claims 1-20; or, to load the computer program to implement the data processing method for immersive media as described in claim 21 or 22.
26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as a data processing method for immersive media as described in any one of claims 1-20; or, loaded and executed as a data processing method for immersive media as described in claim 21 or 22.
27. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and execute the immersive media data processing method as described in any one of claims 1-20; or, to load and execute the immersive media data processing method as described in claim 21 or 22.
Citation Information
Patent Citations
Immersive media acquisition method in multi-degree-of-freedom scene, equipment and storage medium
CN113453083A
