Level Information in Immersive Media Processing

Abstract

A method, apparatus, and system for providing a consistent immersive media viewing experience to a user while reducing bandwidth consumption is disclosed. In one example aspect, a method for processing multimedia content includes: for a conversion between a frame of panoramic media content including a plurality of segments and a bitstream representation of the frame of the panoramic media content, determining a plurality of sets of rating information associated with the frame. Each set of rating information indicates a priority level for processing a segment of the frame of the panoramic media content. The method also includes performing the conversion based on a plurality of rating parameters.

Description

Technical Field

[0001] This patent application is generally directed to multimedia content processing. Background Art

[0002] Users like to experience content the same way they see it in real life. As technology advances, immersive media brings content that is more realistic than ever before. Immersive media includes non-traditional formats driven by emerging technology platforms such as 360-degree images and videos, virtual reality (VR), augmented reality (AR), mixed reality (MR), as well as wearable devices and the Internet of Things (IoT). Summary of the invention

[0003] This patent application describes, among other things, techniques for providing a consistent viewing experience to users while reducing bandwidth consumption for immersive media.

[0004] In one example aspect, a method for processing multimedia content includes: for a conversion between a frame of panoramic media content including a plurality of segments and a bitstream representation of the frame of the panoramic media content, determining a plurality of sets of rating information associated with the frame. Each set of rating information indicates at least a priority level, a quality level, or a stream switch for processing a segment of the panoramic media content frame. The method also includes performing the conversion based on a plurality of rating parameters.

[0005] In another exemplary aspect, an apparatus in a multimedia system is disclosed, wherein the apparatus comprises a processor configured to implement the above method.

[0006] In yet another example aspect, a computer program storage medium is disclosed. The computer program storage medium includes code stored thereon. When executed by a processor, the code causes the processor to implement the described method.

[0007] These and other aspects are described in this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flowchart representation of a method for processing multimedia content according to the present technology.

[0009] Figure 2 Example panoramic images with different values ​​representing level information in accordance with the present technique are shown.

[0010] Figure 3 An example panoramic image is shown with non-contiguous integer values ​​representing level information in accordance with the present technique.

[0011] Figure 4 An example panoramic image with different characters representing level information in accordance with the present technique is shown.

[0012] Figure 5 An example non-uniform partitioning of a panoramic image in accordance with the present technique is shown.

[0013] Figure 6 Another example non-uniform partitioning of a panoramic image in accordance with the present technique is shown.

[0014] Fig. 7A An example panoramic image is shown in which a subset of segments have corresponding level information in accordance with the present technique.

[0015] Figure 7B Another example panoramic image is shown in which a subset of segments have corresponding level information according to the present technique.

[0016] Figure 8 An example cascaded representation of level information in accordance with the present technique is shown.

[0017] Fig. 9 is a block diagram of an example of a video encoding device according to the present technology.

[0018] Fig.10 is a block diagram of an example of a video decoding device according to the present technology.

[0019] Fig.11 is a block diagram of an embodiment of a hardware platform for implementing the methods described herein. DETAILED DESCRIPTION

[0020] In this application, section headings are used only to improve readability and do not limit the scope of the embodiments and techniques disclosed in each section to that section. The disclosed techniques are applicable to various imaging and / or video standards, including but not limited to the High Efficiency Video Coding (HEVC) standard and the Versatile Video Coding (VVC) standard.

[0021] Immersive media refers to the use of multimedia technologies such as audio and video to provide users with an immersive and completely different media experience. When watching immersive media content, users can participate in the media in real time (for example, by wearing a virtual reality helmet). Immersive media has been successfully expanded to many fields such as movies, games, news and / or medical treatment. Immersive media content is different from traditional media content: the visual content is a typical 360-degree panoramic visual data, which covers a large image area and a wide field of view, providing users with more flexibility to adjust the content based on their own preferences. Panoramic visual data can be composed of data collected from multiple cameras at the same time, or by splicing multiple images from a single camera through translation and / or rotation operations. Panoramic visual data can also be artificially synthesized (for example, based on existing traditional and / or panoramic image data). Panoramic visual data can be represented as a 360-degree spherical or mapped rectangular image (for example, an equirectangular projection image). Alternatively, panoramic visual data can be represented as a rectangular image after mapping and rearrangement (for example, a cube mapping projection image).

[0022] Compared with traditional media content, displaying immersive media content requires higher bandwidth. Even with the rapid development of mobile networks, bandwidth demand is still a problem. Therefore, there is still a need to provide an effective way to display immersive media content while maintaining reasonable bandwidth usage.

[0023] Disclosed herein are technologies that can be implemented in various embodiments to provide users with high-quality immersive media content without imposing unreasonable bandwidth requirements on communication systems. Immersive media content provides users with a 360-degree panoramic view, but the field of view of human vision is generally within 120 degrees at any given time. In addition, users may want to focus on very specific features within a narrow field of view. Therefore, it is not necessary to maintain uniform visual quality throughout the panoramic content. Different areas or fragments of panoramic media content can be processed according to a set of priorities to take into account user preferences, scene characteristics, and / or bandwidth allocation, thereby minimizing bandwidth consumption while providing a good user experience. In this patent application, priority is referred to as the level information of visual content. For example, when a user watches an immersive concert video, the user's perspective is mainly focused on the performers on the stage. Therefore, the content that displays the performers can be given a higher level to provide the user with a higher visual quality, while the peripheral view is given a lower level. When the user's viewpoint or the position of the performer changes, the media content can be adjusted accordingly to ensure that the user's viewing area is given a higher priority, thereby providing a consistent viewing experience.

[0024] To facilitate efficient processing of panoramic visual data, level information can be encoded in the bitstream representing the panoramic video content. The growing demand for high-resolution video has led to ubiquitous image and video coding techniques in modern technology. For example, video codecs convert uncompressed video to compressed formats and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access, and end-to-end delay (latency). The compression format usually conforms to a standard video compression specification, such as an ISO / IEC-based media file format, the HEVC standard (also known as H.265 or MPEG-H Part 2), the VVC standard (also temporarily known as H.266 or MPEG-I Part 3), or other current and / or future video coding standards. Many coding standards use hybrid coding such as intra-frame prediction and / or inter-frame prediction coding techniques to eliminate spatial redundancy and temporal redundancy. That is, different areas encoded using predictive coding techniques are related to each other. During video coding, the encoder first divides the image / frame into one or more regions and then encodes these regions. To facilitate parallel processing, a region corresponds to a unit in the image that can be decoded independently. A region can be a slice group in the H.264 / AVC standard, a tile in the H.265 / HEVC standard, a sub-picture in the H.266 / VVC standard, a rectangular slice or a tile.

[0025] Different regions and / or segments of the frame may be prioritized according to different levels to ensure the best user viewing experience. Figure 1 1 is a flowchart representation of a method 100 for processing multimedia content according to the present technology. At operation 110, method 100 includes: for conversion between a frame of panoramic media content including multiple fragments and a bitstream representation of the frame of panoramic media content, determining multiple sets of rating information associated with the frame. Each set of rating information indicates at least a priority level, a quality level, or a stream switch for processing a fragment of the panoramic media content frame. At operation 120, method 100 also includes: performing the conversion based on multiple rating parameters. The conversion between the frame of panoramic media content and the bitstream representation includes an encoding and / or decoding process. When encoding, the rating information can be encoded in the bitstream when the panoramic media content is collected in real time and / or collected based on the characteristics of the content and user preferences.

[0026] In some embodiments, each set of rating information includes an integer indicating a priority level for processing a corresponding segment of the panoramic media content. In some embodiments, multiple sets of rating information include consecutive integers indicating priority levels for processing multiple segments of the panoramic media content. In some embodiments, each set of rating information includes one or more characters indicating a priority level for a corresponding segment of the panoramic media content.

[0027] In some embodiments, the multiple segments within the frame are of equal size. For example, the multiple segments all have a rectangular shape. In some embodiments, at least one segment has a different size than another segment within the frame. For example, at least one segment has a non-rectangular shape.

[0028] In some embodiments, the bitstream representation includes a parameter indicating a number of sets of level information. In some embodiments, each set of level information includes one or more parameters indicating a dimension of a corresponding segment. In some embodiments, at least one segment within a frame lacks a corresponding set of level information.

[0029] In some embodiments, the bitstream representation comprises timing information indicating when the multiple sets of level information are applicable to a frame. In some embodiments, the bitstream representation comprises view information indicating one or more views at which the multiple sets of level information are applicable to the frame.

[0030] In some embodiments, where the first priority level has a higher value than the second priority level, a first fragment corresponding to the first priority level is processed before a second fragment corresponding to the second priority level. In some embodiments, only a subset of the fragments are processed for conversion. The priority levels of the subset of fragments may have a value equal to or greater than a predefined threshold. In some embodiments, the priority levels of the subset of fragments are included in the bitstream representation based on the position of the subset of fragments within the frame. In some embodiments, each set of level information includes level information for multiple sub-segments within the corresponding fragment.

[0031] In some embodiments, the method further comprises: determining, for a conversion between a subsequent frame of the panoramic media content and the bitstream representation, an additional set of rating information associated with a subsequent frame of the panoramic media content. In some embodiments, the additional sets of rating information associated with the subsequent frame include values ​​relative to the sets of rating information associated with the frame. In some embodiments, at least one of the sets of rating information associated with the frame is applicable to subsequent frames of the panoramic media content.

[0032] In some embodiments, the region comprises a coding tree unit, a coding tree block, a tile, a macroblock, or a sub-picture. In some embodiments, the multiple sets of level information are represented in supplemental enhancement information or video usability information in the bitstream representation. In some embodiments, the bitstream representation of the panoramic media content is formatted according to ISO / IEC 14496.

[0033] In some embodiments, performing the conversion includes generating a bitstream representation based on the current block of video. In some embodiments, performing the conversion includes generating the current block of video from the bitstream representation.

[0034] Some examples of the disclosed technology are further described in the following exemplary embodiments.

[0035] Example 1

[0036] The level information may be represented in different forms in the bitstream. In some embodiments, the level information may be represented using one or more numerical values. Figure 2 An example panoramic image 200 is shown with different values ​​representing level information in accordance with the present technique. Figure 2 In , the panoramic image is divided into 9 rectangular regions (or segments). Each region is assigned a value ranging from 1 to 3. Different values ​​can represent different image quality levels and / or different processing priorities of the corresponding regions. For example, 1 represents high image quality, 2 represents medium image quality, and 3 represents low image quality. As another example, 1 represents the highest processing priority, which indicates that the corresponding region is processed first in the encoding / decoding process. 2 represents medium processing priority, and 3 represents the lowest processing priority.

[0037] In some embodiments, the value of the level information can represent clips of different directors. For example, 1 represents the clip of the first director, and 2 represents the clip of the second director. The user can view different pictures by selecting different level values ​​corresponding to different clips. When processing the video bitstream, the decoder can parse and determine the level information of the area. Then, the decoder obtains the subsequent video stream of the area based on the level information. For example, when the user chooses to watch the clip of the first director (for example, the level value is 1), the decoder obtains the video stream at time T, and obtains the stream corresponding to the level value 1 to display the clip of the first director. When the user chooses to watch the clip of the second director (for example, the level value is 2), the decoder obtains the video stream at time T, and obtains the subsequent stream corresponding to the level value 2 to display the clip of the second director.

[0038] In some embodiments, the value of the level information may be a non-consecutive integer. Figure 3 An example panoramic image 300 is shown with non-contiguous integer values ​​representing level information in accordance with the present technique. Figure 3As shown, non-consecutive integer values ​​23, 26, and 30 are used to indicate different quality and / or priority levels for corresponding regions in the panoramic image.

[0039] In some embodiments, the level information may include scene features represented using one or more characters. Figure 4 An example panoramic image 400 is shown with different characters representing level information according to the present technology. Figure 4 As shown, different characters can represent the attributes or scene features of the associated area. For example, the character "P" represents a person, indicating that the corresponding area displays one or more people. The character "C" represents a car, indicating that the corresponding area displays one or more cars. Similarly, the character "B" represents a background, indicating that the corresponding area displays a scene background. In some embodiments, a string containing multiple characters can also be used.

[0040] Example 2

[0041] The regions / segments in the panoramic image may have different sizes and / or shapes. Figure 2-4 As shown, in some embodiments, the panoramic image can be equally divided into a plurality of regions / segments. In some embodiments, the panoramic image can be divided into regions of non-uniform shape. Figure 5 An example non-uniform partitioning of a panoramic image 500 according to the present technique is shown. In this example, the regions / segments are rectangular areas of non-uniform size. Figure 6 Another example non-uniform partitioning of a panoramic image 600 according to the present technology is shown. In this example, the regions / segments have non-rectangular shapes. At least one segment is different in size and / or shape from another segment in the panoramic image.

[0042] Example 3

[0043] As described above, in order to speed up the processing of panoramic images (e.g., on the decoding side), the level information can be encoded in the bitstream representation to reduce additional calculation and / or processing time. The level information can be organized according to regions / segments. Table 1 shows an example representation of the level information in the bitstream according to the present technology.

[0044] Table 1 Example representation of level information

[0045]

[0046] Here, num_ranks indicates the number of rank information. This value is the same as the number of segments in the panoramic image.

[0047] Table 2 shows another example representation of level information in a bitstream according to the present technique.

[0048] Table 2 Another example representation of level information

[0049]

[0050] Here, rank_cnt_minus1 indicates the number of rank information (eg, rank_cnt_minums1+1). This value is the same as the number of segments in the panoramic image.

[0051] In Tables 1 and 2, segment_top_left_x[i] indicates the horizontal coordinate of the top left pixel of the i-th segment in the panoramic image. segment_top_left_y[i] indicates the vertical coordinate of the top left pixel of the i-th segment in the panoramic image. segment_width[i] indicates the pixel width of the i-th segment. segment_height[i] indicates the pixel height of the i-th segment. segment_rank[i] indicates the rank value of the i-th segment.

[0052] In some embodiments, the coordinates of the center point of the segment can be used as the anchor point of the segment, rather than the coordinates of the upper left point of the segment. In some embodiments, the coordinates and anchor points of the segment can be adjusted for 360-degree spherical panoramic 3D coordinates (e.g., azimuth range, elevation range).

[0053] The fragments disclosed herein may be associated with different partition units based on different standards. For example, a fragment may be a grid unit such as a coding tree unit (CTU) in the HEVC / H.265 standard. In the HEVC / H.265 standard, a coding tree block (CTB) is an NxN block of luminance and / or chrominance samples, where N can be 16, 32, 64, etc. A CTU includes a luminance CTB and two chrominance CTBs, and is an independent coding unit. The CTU in VVC / H.265 is consistent with the CTU concept in HEVC / H.265, and is also an independent coding unit. In the AVC / H.264 standard, a grid unit may also be a macroblock. A macroblock is a 16x16 luminance block and its corresponding chrominance block. In the AVC / H.264 standard, a macroblock is an independent coding unit.

[0054] Different standards partition frames in different ways. For example, in the HEVC / H.265 and VVC / H.266 standards, partitions can be represented in the form of tiles. Tiles divide an image into several rectangular areas horizontally and vertically. A tile is a rectangular area of ​​a CTU within a specific tile column and a specific tile row in a picture. One or more tiles may correspond to a partition of an image. In the HEVC / H.265 standard, partitions may also be represented by slices. One or more slices correspond to a partition and include multiple rows of CTUs. In the VVC / H.266 standard, partitions may be represented in the form of raster scan slices. One or more raster scan slices correspond to a partition and include multiple rows of CTUs. Partitions may also be represented by rectangular slices. One or more rectangular slices may correspond to a partition. Partitions may also be represented by sub-pictures. One or more sub-pictures may correspond to a partition.

[0055] Table 3 shows another example representation of level information in a bitstream according to the present technique.

[0056] Table 3 Another example representation of level information

[0057]

[0058]

[0059] Table 4 shows another example representation of level information in a bitstream according to the present technique.

[0060] Table 4 Another example representation of level information

[0061]

[0062] The rank information of each segment may be arranged in a specific predefined manner (eg, zigzag scan order, ascending order, or descending order). In some embodiments, the rank information may be randomly arranged in any order.

[0063] In some embodiments, the rank information may exclude specific information about the segments, such as the anchor point and / or size of the segments. Table 5 shows another example representation of rank information in a bitstream according to the present technology. As shown in Table 5, no specific information about the segments is included in the RankMap. Table 6 shows another example representation of rank information in a bitstream according to the present technology. Similarly, no specific information about these segments is included.

[0064] Table 5 Another example representation of level information

[0065]

[0066]

[0067] Table 6 Another example representation of level information

[0068]

[0069] Specific information about the segment may be located separately in the bitstream (eg, before the level information). In some embodiments, a default segment may be used to eliminate the need to include such information in the bitstream.

[0070] In some embodiments, the segment corresponding to the rank information may be a sub-picture defined in the VVC / H.266 standard. The segment position and size information is the position and size information of the sub-picture. In these cases, rank_cnt_minus1 and sps_num_subpics_minus1 are equal. Each segment_rank[i] and sps_subpic_id[i] have a one-to-one correspondence.

[0071] Table 7 Another example representation of level information

[0072]

[0073] In some embodiments, the segment corresponding to the level information may be a Temporal Motion Constrained Tile Set (MCTS) defined in the H.265 / HEVC standard. Table 8 shows an example representation of the level information as part of the MCTS according to the present technology.

[0074] Table 8 Another example representation of level information

[0075]

[0076] Example 4

[0077] In some embodiments, the rank information includes specific information only for a subset of the segments. For example, num_ranks as shown in Tables 1 and 2 may be less than the total number of segments in the panoramic image. Fig. 7A An example panoramic image 700 is shown in which a subset of segments have corresponding level information in accordance with the present technique. Fig. 7A As shown, only the nine segments in the lower right corner of the panoramic image have corresponding level information. Note that the subsets of segments do not need to be adjacent to each other. Figure 7B Another example panoramic image 750 is shown in which a subset of segments have corresponding level information in accordance with the present technique.

[0078] In some embodiments, a default level value may be used to process remaining fragments that do not have specific level information.

[0079] Example 5

[0080] In some embodiments, the level information is associated with time. For example, the level information is only applicable to the processing of the panoramic image within a certain duration. Table 9 shows an example representation of the level information in a bitstream with corresponding time information according to the present technology.

[0081] Table 9 Another example representation of level information

[0082]

[0083] The variable time_flag equals 1 to indicate that there is time information associated with the level information. Its value of 0 indicates that there is no relevant time information. TimeInfoStruct may include information describing the effective time interval of the level information. For example, TimeInfoStruct may include an absolute start time and an end time. As another example, an absolute start time and a duration time interval may be used. In some embodiments, a relative start / end time about the last effective level information may be indicated (see details in Embodiment 7 described below).

[0084] If new level information is obtained within the duration for which the level information is valid, the new level information may be adopted according to preset rules. Alternatively, the new level information may be ignored until the duration of the current level information ends. Table 10 shows an example representation of level information in a bitstream with corresponding time information according to the present technology. The variable rank_map_info_cancel flag is 1 to indicate that the current level information is retained; its value is 0 to indicate that the new level information is adopted. The variable rank_map_info_persistence_flag is 0 to indicate that the current level information is only applicable to the current frame / image; its value is 1 to indicate that the current level information is applicable to subsequent frames / images.

[0085] Table 10 Another example representation of level information

[0086]

[0087] Example 6

[0088] In some embodiments, the level information is associated with the user's viewpoint. Table 11 shows an example representation of level information in a bitstream with corresponding temporal information according to the present technique.

[0089] Table 11 Another example representation of level information

[0090]

[0091] (viewpoint_pos_x, viewpoint_pos_y, viewpoint_pos_z) is the position of the viewpoint. The viewpoint position can also use GPS coordinate information (longitude, latitude, altitude), and the viewpoint position can also be a three-dimensional polar coordinate (yaw, pitch, roll). The variable rankmap_flag is 1 to indicate that there is rank information; its value is 0 to indicate that there is no rank information.

[0092] The level information may also change with the switching of viewpoints. Table 12 shows an example representation of the level information in a bitstream according to the present technique.

[0093] Table 12 Another example representation of level information

[0094]

[0095] Here, num_viewpoint_switching indicates the number of viewpoint switches, and destination_viewpoint_id indicates the sequence number of the destination viewpoint. The variable timeline_switching_offset_flag is 1 to indicate that there is switching time information; its value is 0 to indicate that there is no switching time information. The variable rankmap_flag is 1 to indicate that there is rank information; its value is 0 to indicate that there is no rank information. ViewpointTimelineSwitchStruct indicates the time point of view switching.

[0096] In some embodiments, the level information may not change every time the viewpoint changes. If the viewpoint switch is not associated with a specific set of level information, the currently valid level information or the default predefined level information may be used.

[0097] Example 7

[0098] The rank value in the rank information can be an absolute value or a relative value. Table 13 shows an example representation of rank information in a bitstream according to the present technology. The variable rank_diff_flag can be added to indicate whether an absolute value or a relative value is used. When the value is 1, a relative rank value can be used. When the value is 0, an absolute value can be used.

[0099] Table 13 Another example representation of level information

[0100]

[0101] Example 8

[0102] The level information can be represented in a cascade form. Figure 8 An example cascaded representation 800 of level information according to the present technique is shown. Figure 8 The thick solid lines shown represent segments, while the thin solid lines represent sub-segments. For example, 11 represents 1 subdivision of segment 1. Table 14 shows an example representation of level information in a bitstream according to the present technique.

[0103] Table 14 Another example representation of level information

[0104]

[0105] Here, sub_segment_cnt_minus1 indicates the number of sub-segment rank information. sub_segment_top_left_x[j] is the coordinate of the top left pixel of the j-th sub-segment in the horizontal direction of the image. sub_segment_top_left_y[j] is the vertical coordinate of the top left pixel of the j-th sub-segment in the image. sub_segment_width[j] is the pixel width of the j-th sub-segment. sub_segment_height[j] is the pixel height of the j-th sub-segment. sub_segment_rank[j] is the rank value of the j-th sub-segment.

[0106] In some embodiments, the segment does not have a corresponding rank value (eg, segment_rank[i] information does not exist). Only sub-segment rank values ​​are used.

[0107] Example 9

[0108] In some embodiments, the level information is represented in supplemental enhancement information (SEI) in the video stream. Table 15 shows an example representation of level information in a bitstream according to the present technique.

[0109] Table 15 Another example representation of level information

[0110]

[0111] In some embodiments, the rank information is represented in the video usability information (VUI) in the video bitstream. Table 16 shows an example representation of rank information in a bitstream according to the present technology. The variable rank_map_info_flag can indicate whether rank_map_info is used. When the value is 1, rank_map_info can be used. When the value is 0, rank_map_info can be skipped.

[0112] Table 16 Another example representation of level information

[0113]

[0114] Example 10

[0115] The media code stream in the above embodiment may be stored in a media file, for example, based on the International Organization for Standardization (ISO) basic media file format.

[0116] All data in the ISO base file format is packaged in boxes. That is, the ISO base file format represented by the MP4 file consists of several boxes, each of which has a type and length and can be regarded as a data object. A box can contain another box, which is called a container box. The MP4 file has only one box of type "ftyp", which serves as a marker of the file format and contains some information about the file. There is also only one box of type "MOOV" (movie box), which is a container box. The child boxes of the movie box include metadata information of the media. The media data of the MP4 file is included in the "mdat" type box (media data box), which is also a container box. In addition, the timed metadata track is a mechanism in the ISO base media file format (ISOBMFF) for establishing timed metadata associated with a specific sample.

[0117] For example, Table 17 shows an example viewpoint using a track entry type of "vipo". The level information can be combined with the viewpoint information (eg, located in ViewPointPostStruct or ViewpointSwitchingStruct).

[0118] Table 17 Viewpoint definition example

[0119]

[0120] Fig. 99 is a block diagram of an example of a video encoding device 900 according to the present technology. The device 900 includes an acquisition module 901, which is configured to collect spherical panoramic video image information. This module may include an input-output controller circuit for reading video data from a memory or a camera frame buffer. This module may include processor-executable instructions for reading video data. The device 900 includes a level information processing module 902, which is configured to determine the level information associated with the segment / region. This module can be implemented as a processor-executable software code. The device 900 also includes a video encoder module 903, which is configured to encode the video image data according to the level information. The video encoder module 903 can be, for example, a traditional H.264 / H.265 or another codec suitable for encoding videos and images of rectangular shapes. The video encoder module 903 can use a technique using motion estimation / compensation or intra-image coding techniques. The apparatus further includes a storage / transmission module 904 configured to perform storage or network transmission layer encoding on the video encoding data or the media data.

[0121] The acquisition device 901, the level information processing module 902, the video encoder 903 and the transmission module 904 described above can be implemented by using dedicated hardware or hardware capable of performing processing in conjunction with appropriate software. Such hardware or dedicated hardware may include application-specific integrated circuits (ASICs), various other circuits, various processors, etc. When implemented by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or multiple independent processors, some of which may be shared. In addition, a processor should not be understood to refer to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage devices.

[0122] like Fig. 9 The illustrated apparatus 900 may be a device in a video application such as a mobile phone, a computer, a server, a set-top box, a portable mobile terminal, a digital camera, a television broadcasting system device, and the like.

[0123] Fig.101 is a block diagram of an example of a video decoding device 1000 according to the present technology. The device 1000 includes an acquisition module 1001, which is configured to obtain a bit stream from a network or a storage device, and perform decoding from network transport layer data including video encoding data or media data, and can only extract regional adjacency information data, or extract video data including regional adjacency information. The level information processing module 1002 is configured to parse the level information. The video decoder 1003 is configured to decode video data information, decode panoramic video, or extract an independent decoding unit where an area of ​​interest is located for decoding. The display device 1004 is configured to generate or display a panoramic video image or select part of an adjacent area image for display.

[0124] The above-mentioned transmission module 1001, area adjacent information processing module 1002, video decoder 1003 and display device 1004 can be implemented by using dedicated hardware or hardware capable of performing processing in combination with appropriate software. Such hardware or dedicated hardware may include application-specific integrated circuits (ASICs), various other circuits, various processors, etc. When implemented by a processor, the function may be provided by a single dedicated processor, a single shared processor, or multiple independent processors, some of which may be shared. In addition, a processor should not be understood to refer to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage devices.

[0125] like Fig.10 The device shown may be a device in a video application such as a mobile phone, a computer, a server, a set-top box, a portable mobile terminal, a digital video camera, a television broadcasting system device, and the like.

[0126] Fig.11 An example device 1100 is shown that can be used to implement the encoder-side or decoder-side techniques described in the present application. The device 1100 includes a processor 1102, which can be configured to perform the encoder-side or decoder-side techniques or both. The device 1100 may also include a memory (not shown) for storing processor-executable instructions and for storing video bitstreams and / or display data. The device 1100 may include video processing circuits (not shown) such as transform circuits, arithmetic encoding / decoding circuits, etc. The video processing circuits may be partially included in the processor and / or partially included in other dedicated circuits, such as a graphics processor, a field programmable gate array (FPGA), etc.

[0127] It should be understood that the present application discloses techniques that can be implemented in various embodiments to provide users with a consistent immersive content viewing experience while reducing bandwidth requirements. The disclosed media content processing system and other embodiments, modules, and functional operations described in the present application can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in the present application and their structural equivalents), or in a combination of one or more of them. The disclosed embodiments and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium, which are used to be executed by a data processing device or control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition of matter that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment, and machines for processing data, such as a programmable processor, a computer, or multiple processors or multiple computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, eg, a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0128] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages), and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed for execution on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0129] The processes and logic flows described in this application can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, for example, FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0130] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data to receive data from the mass storage device or to transfer data to the mass storage device, or both. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM optical disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[0131] Although this patent application contains many details, these details should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this patent application in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations, and even initially claimed to be so protected, in some cases, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0132] Likewise, although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, in order to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent application should not be understood as requiring such separation in all embodiments.

[0133] Only some implementations and examples are described, and other implementations, improvements and variations may be made based on what is described and illustrated in this patent application.

Claims

1. A method for processing multimedia content, comprising: For a first conversion between a first frame of panoramic media content including a plurality of fragments and a bitstream representation of the panoramic media content, determining a plurality of sets of rating information associated with the first frame, wherein each set of rating information of the plurality of sets of rating information indicates at least a priority level, a quality level, or a stream switch for processing a fragment of the first frame of the panoramic media content; determining, for a second transition between a second frame of panoramic media content that temporally follows the first frame and the bitstream representation, an additional set of rating information associated with the second frame of panoramic media content, wherein the additional set of rating information associated with the second frame includes values ​​relative to the multiple sets of rating information associated with the first frame; and Based on the multiple sets of level information, the first conversion is performed.

2. The method according to claim 1, wherein: Each set of the plurality of sets of level information includes an integer indicating a priority level for processing a corresponding segment of the panoramic media content.

3. The method according to claim 2, wherein: The multiple sets of level information include continuous or discontinuous integers, and the continuous or discontinuous integers indicate priority levels for processing multiple segments of the panoramic media content.

4. The method according to claim 1, wherein: Each set of the plurality of sets of rating information includes one or more characters indicating a priority level for a corresponding segment of the panoramic media content.

5. The method according to any one of claims 1 to 4, wherein: The multiple segments within the first frame are equal in size.

6. The method according to claim 5, wherein: The plurality of segments have a rectangular shape.

7. The method according to any one of claims 1 to 4, wherein: At least one segment has a different size than another segment within the first frame.

8. The method according to claim 7, wherein: The at least one segment has a non-rectangular shape.

9. The method according to any one of claims 1 to 4, wherein: The bitstream representation comprises a parameter indicating the number of the plurality of sets of level information.

10. The method according to any one of claims 2 to 4, wherein: Each set of the plurality of sets of level information comprises one or more parameters, where the one or more parameters indicate a dimension of the corresponding segment.

11. The method according to any one of claims 1 to 4, wherein: At least one segment within the first frame lacks a corresponding set of level information.

12. The method according to any one of claims 1 to 4, wherein: The bitstream representation includes timing information indicating when the sets of level information are applicable to the first frame.

13. The method according to any one of claims 1 to 4, wherein: The bitstream representation comprises view information indicating one or more views at which the sets of level information are applicable to the first frame.

14. The method according to any one of claims 1 to 4, wherein: In case a first priority level has a higher value than a second priority level, a first fragment corresponding to the first priority level is processed before a second fragment corresponding to the second priority level.

15. The method according to any one of claims 1 to 4, wherein: Only a subset of the fragments are processed for the first conversion, and wherein the priority levels of the subset of fragments have a value equal to or greater than a predefined threshold.

16. The method according to any one of claims 1 to 4, wherein: Only a subset of the segments are processed for the first conversion, and wherein a priority level of the subset of segments is included in the bitstream representation based on a location of the subset of segments within the first frame.

17. The method according to any one of claims 1 to 4, wherein: Each set of level information of the plurality of sets of level information includes level information for a plurality of sub-segments within a corresponding segment.

18. The method according to claim 1, wherein: At least one of the multiple sets of rating information associated with the first frame is applicable to a second frame of the panoramic media content.

19. The method according to any one of claims 1 to 4, wherein: A region includes a coding tree unit, a coding tree block, a tile, a macroblock or a sub-picture.

20. The method according to any one of claims 1 to 4, wherein: The multiple sets of level information are represented in supplemental enhancement information or video usability information in the bitstream representation.

21. The method according to any one of claims 1 to 4, wherein: The bitstream representation of the panoramic media content is formatted according to ISO / IEC 14496.

22. The method according to any one of claims 1 to 4, wherein: Performing the first conversion includes generating the bitstream representation based on a current block of video.

23. The method according to any one of claims 1 to 4, wherein: Performing the first conversion includes generating a current block of video from the bitstream representation.

24. An apparatus in a multimedia system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method according to any one of claims 1 to 23.

25. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising a program code, wherein the program code implements the method according to any one of claims 1 to 23 when executed by a processor.

Images