Information processing apparatus and method

Abstract

This disclosure relates to an information processing apparatus and method capable of suppressing increased rendering processing load. The invention generates a file storing: a bitstream track containing the content of sub-images within a storage frame; and an extraction base track storing sub-image specification information, which specifies the sub-images to be extracted from the bitstream track. Based on the extraction base track in the file, the invention extracts sub-images from the bitstream track and generates a bitstream. This invention can be applied, for example, to information processing apparatus or information processing methods.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus and methods, and more specifically, to information processing apparatus and methods capable of suppressing an increase in the load of reproduction processing. Background Technology

[0002] Typically, in the general video codec (VVC) of standard specifications for image codec systems, each image (picture) is divided into one or more slices. Furthermore, each picture can also be divided into one or more subpictures (see, for example, Non-Patent Document 1). Additionally, as a method for storing files containing pictures encoded by VVC, a VVC file format using the International Organization for Standardization Basic Media File Format (ISOBMFF), an international standard technology for video compression, is being developed, along with the file container specification for Moving Picture Experts Group-4 (MPEG-4) (see, for example, Non-Patent Documents 2 to 4).

[0003] Furthermore, there exists video-based point cloud compression (V-PCC), in which a point cloud, which is a set of points possessing both positional and attribute information (color, reflection, etc.) in three-dimensional space, is segmented to form regions. Attribute images (color information, etc.) and geometric images (composed of depth information) are generated by planar projection of the point cloud into each region. Eye diagram images and atlas information (information used to reconstruct the point cloud from image blocks) are also generated, and the three images are encoded by a motion picture codec. Then, to improve the efficiency of reproduction processing and network distribution from the local storage of the bitstream encoded by V-PCC (referred to as the V3C bitstream), a technique for storing the V3C bitstream in ISOBMFF has been proposed (see, for example, non-patent literature 5).

[0004] In the VVC file format described above, multiple sub-images can be stored in a single track (VVC track) and reproduced.

[0005] Reference List

[0006] Non-patent literature

[0007] Non-Patent Document 1: Benjamin Bross, Jianle Chen, Shan Liu, Ye-Kui Wang, “Versatile Video Coding (Draft 10)”, JVET-S2001-vH, ITU-T SG 16WP3 and ISO / IEC JTC1 / SC 29 / WG 11, 19th Teleconference of the Joint Video Experts Group (JVET): June 22 to July 1, 2020;

[0008] Non-patent document 2: "Information technology - Coding of audio-visual objects - Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format - Amendment 2: Carriage of VVC and EVC in ISOBMFF", ISO / IEC JTC 1 / SC 29 / WG 11, ISO / IEC 14496-15:2019(E) Revision 2, July 30, 2020;

[0009] Non-patent document 3: "Information technology - Coding of audio-visual objects - Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format", ISO / IEC JTC 1 / SC 29 / WG 11, September 2019;

[0010] Non-patent document 4: "Information technology - Coding of audio-visual objects - Part 12: ISO base media file format", ISO / IEC JTC 1 / SC 29 / WG 11, December 2015;

[0011] Non-patent document 5: "Information technology - Coded representation of immersive media - Part 10: Carriage of Visual Volumetric Video-based Coding Data", ISO / IEC JTC1 / SC 29 / WG 11, ISO 23090-10:2020(E), August 23, 2020. Summary of the Invention

[0012] The problem to be solved by the present invention

[0013] However, in order to reproduce some sub-images in the VVC track, it is necessary to analyze and generate the parameter set of the VVC bitstream, and there is a possibility of increased processing overhead for reproduction.

[0014] The present invention was made in view of this situation, and its purpose is to suppress the increase in the load of the reproduction process.

[0015] Solution to the problem

[0016] An information processing apparatus of one aspect of this technology is an information processing apparatus comprising: a file generation unit configured to generate a file storing a bitstream track and extracting a base track, the bitstream track being stored as a bitstream containing sub-pictures within a frame, and the base track being stored as sub-picture specification information, the sub-picture specification information being information used to specify the sub-pictures to be extracted from the bitstream track.

[0017] One aspect of the information processing method of this technology is as follows: generating a file for storing bitstream tracks and extracting basic tracks, wherein the bitstream tracks store bitstreams containing sub-images within frames, and the extracted basic tracks store sub-image specified information, wherein the sub-image specified information is information used to specify the sub-images to be extracted from the bitstream tracks.

[0018] Another aspect of the information processing apparatus of this technology is the following information processing apparatus, which includes: a bitstream generation unit configured to extract a basic track based on a file storing a bitstream track and extracting a basic track, extract sub-pictures from the bitstream track and generate a bitstream, wherein the bitstream track is stored as a bitstream containing sub-pictures within a frame, and the extracting basic track stores sub-picture specification information, which is information used to specify the sub-pictures to be extracted from the bitstream track.

[0019] Another aspect of the information processing method of this technology is as follows: extracting a basic track based on a file storing a bitstream track and extracting a basic track; extracting sub-images from the bitstream track and generating a bitstream; storing a bitstream containing sub-image content within a frame in the bitstream track; and storing sub-image specified information in the extracted basic track, wherein the sub-image specified information is information used to specify the sub-images to be extracted from the bitstream track.

[0020] In an information processing apparatus and method according to one aspect of the present technology, a file for generating and storing bitstream tracks and extracting basic tracks is generated. The bitstream tracks store bitstreams containing sub-images within frames, and the extracted basic tracks store sub-image specification information, which is information used to specify the sub-images to be extracted from the bitstream tracks.

[0021] In another aspect of the information processing apparatus and method of this technology, a sub-image is extracted from a file based on a stored bitstream track and an extracted base track, and a bitstream is generated. The bitstream track stores a bitstream containing the content of the sub-image within a frame. The extracted base track stores sub-image specification information, which is information used to specify the sub-image to be extracted from the bitstream track. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating a primary configuration example of a VVC bitstream.

[0023] Figure 2 This is a diagram illustrating an example of the syntax for a sequence parameter set.

[0024] Figure 3 This is a diagram illustrating an example of the syntax for image parameter sets.

[0025] Figure 4 This is a diagram illustrating an example of the syntax for image parameter sets.

[0026] Figure 5 This is a diagram showing a primary configuration example of a VVC track.

[0027] Figure 6 This is a diagram showing an example of a sequential sample group of sub-images.

[0028] Figure 7 This is a diagram showing an example of a sample group of sub-image IDs.

[0029] Figure 8 This is a diagram showing the tiled area.

[0030] Figure 9 This is a diagram showing an example of a sample group of tile regions.

[0031] Figure 10 This is a diagram showing an example of a NAL cell mapping sample group.

[0032] Figure 11 This is a diagram illustrating use cases for sub-images.

[0033] Figure 12 This is a diagram illustrating an example of the V3C file format.

[0034] Figure 13 This diagram illustrates a method for using VVC to extract basic orbital information and signal it for reconstruction.

[0035] Figure 14 This is a diagram illustrating an example configuration for storing files that extract the base track from VVC.

[0036] Figure 15This is a diagram showing an example of extracting sample groups from sub-images.

[0037] Figure 16 This is a diagram showing an example of a sample group of sub-image IDs.

[0038] Figure 17 This is a diagram showing an example of a sample group of sub-image IDs.

[0039] Figure 18 This is a diagram showing the combination of the VVC basic track and the VVC extracted basic track.

[0040] Figure 19 This is a diagram showing an example of a sequential sample group of sub-images.

[0041] Figure 20 This is a diagram illustrating an application example in the case of V3C bitstreams.

[0042] Figure 21 This is a block diagram illustrating a primary configuration example of a file generation device.

[0043] Figure 22 This is a flowchart illustrating an example of the file generation process.

[0044] Figure 23 This is a block diagram illustrating a primary configuration example for a client device.

[0045] Figure 24 This is a flowchart illustrating an example of the reproduction process.

[0046] Figure 25 This is a block diagram illustrating a typical configuration example of a computer. Detailed Implementation

[0047] The following describes the manner in which this disclosure is carried out (hereinafter referred to as implementation). Note that the description will be given in the following order.

[0048] 1. Reproduction of partial sub-images

[0049] 2. Extract the basic track

[0050] 3. Expansion of Sub-image ID Sample Groups

[0051] 4. Combination of VVC basic tracks and VVC additional basic tracks

[0052] 5. Application of V3C file format

[0053] 6. First Implementation Method (Document Generation Device)

[0054] 7. Second embodiment (reproduction device).

[0055] 8. Appendix

[0056] <1. Reproduction of partial sub - pictures>

[0057] <Documents supporting technical content, technical terms, etc.>

[0058] The scope disclosed in the present technology includes not only the content described in the embodiments, but also the content described in the following non - patent documents, etc. known at the time of filing, the content of other documents cited in the following non - patent documents, etc.

[0059] Non - patent document 1: (described above);

[0060] Non - patent document 2: (described above);

[0061] Non - patent document 3: (described above);

[0062] Non - patent document 4: (described above);

[0063] Non - patent document 5: (described above).

[0064] That is, the content described in the above non - patent documents, the content of other documents referred to in the above non - patent documents, etc. are also the basis for determining the support requirements.

[0065] <VVC bitstream>

[0066] Traditionally, in the Versatile Video Coding (VVC) of the standard specification of image coding and decoding methods, each image (picture) is divided into one or more slices, for example, as described in Non - patent document 1. In addition, each picture can also be divided into one or more sub - pictures.

[0067] Figure 1 is a diagram showing a main configuration example of a bitstream (also referred to as a VVC bitstream) obtained by encoding image data by VVC. As Figure 1 shown, the VVC bitstream 10 includes a set (single or multiple decoded video sequences 11) of decoded video sequences 11 obtained by encoding a moving image. In addition, the decoded video sequence 11 includes a set (single or multiple encoded data of pictures 12) of encoded data of pictures 12 that are single frames of the moving image. Each picture 12 can be divided into multiple sub - pictures 13. <(

[0068] That is, the sub - picture 13 is a rectangular area obtained by dividing the rectangular picture 12. There are no pixels without encoded data in the picture 12. There is no overlap between the sub - pictures 13. Among the pixels of the picture 12, there are no pixels that are not included in any sub - picture 13.

[0069] Sub-image 13 can be independently decoded. That is, one or more sub-images 13 can be decoded independently. For example, only a portion of the encoded data of sub-image 13 that constitutes image 12 can be decoded. In this case, the encoded data of the sub-image 13 to be decoded is extracted from the VVC bitstream 10, and parameter sets, etc., are added to generate a new bitstream (the bitstream is reconstructed).

[0070] Figure 2 An example of the syntax for the sequence parameter set of VVC bitstream 10 is shown. The sequence parameter set is a set of parameters (metadata that does not change in the encoded video sequence 11) used for each encoded video sequence 11. Figure 2 The syntax for a portion of the sequence parameter set is shown.

[0071] like Figure 2 As shown, the sequence parameter set includes "picture resolution information" configured by parameters related to the resolution of picture 12. For example, this "picture resolution information" includes parameters (sps_pic_width_max_in_luma_samples, sps_pic_height_max_in_luma_sample, etc.) indicating the maximum value of the resolution of picture 12 in the encoded video sequence 11.

[0072] Furthermore, the sequence parameter set includes "subpic layout information" configured by parameters related to the layout of subpic 13. For example, "subpic layout information" includes parameters indicating the position of subpic 13 (sps_subpic_ctu_top_left_x[i], sps_subpic_ctu_top_left_y[i], etc.). Additionally, "subpic layout information" includes parameters indicating the resolution of subpic 13 (sps_subpic_width_minus1[i], sps_subpic_height_minus1[i], etc.). Furthermore, "subpic layout information" includes parameters related to the independent decoding of subpic 13 (sps_subpic_treated_as_pic_flag[i], sps_loop_filter_across_subpic_enabled_flag[i], etc.).

[0073] Furthermore, the sequence parameter set includes "subpic layout omission flags" configured with information about the omission of "subpic layout information". For example, the "subpic layout omission flags" include flags indicating whether all subpicks 13 can be decoded independently (sps_independent_subpic_flag), flags indicating whether all subpicks 13 have the same resolution (sps_subpic_same_size_flag), etc. For example, if all subpicks 13 can be decoded independently and have the same resolution, regardless of the number of subpicks 13, a parameter related to the width of the subpicks 13 and a parameter related to the height of the subpicks 13 are stored in the sequence parameter set.

[0074] Furthermore, the sequence parameter set includes "subpic identification information" configured by parameters related to the identification of subpic 13. For example, "subpic identification information" includes the identification information (sps_subpic_id[i]) of each subpic 13 included in the encoded video sequence. For example, a number can be sequentially assigned to each subpic in the sequence (e.g., assigning an integer value incrementing by "+1", such as 0, 1, 2, ...), and this number can be used as the identification information of subpic 13. Note that there are three types of "subpic identification information": the case where the subpic identification information is omitted, the case where the subpic identification information is stored in the sequence parameter set, and the case where the subpic identification information is stored in the picture parameter set.

[0075] When extracting and decoding sub-images 13 from such a VVC bitstream 10, the "sub-image identification information" and "sub-image layout information" may change because the number of sub-images included in the encoded / decoded video sequence 11 changes. That is, in this case, it is necessary to update the sequence parameter set of the VVC bitstream 10 (encoded / decoded video sequence 11) and generate a sequence parameter set corresponding to the bitstream of the extracted sub-images.

[0076] also, Figure 3 and Figure 4 It shows Figure 1 This is an example of the syntax for the image parameter set in VVC bitstream 10. The image parameter set is the set of parameters used for each image 12 (metadata that does not change in image 12). Figure 3 and Figure 4 The syntax for a portion of the image parameter set is shown in the image.

[0077] like Figure 3As shown, the picture parameter set includes "picture resolution information" configured by parameters related to the resolution of picture 12. For example, the "picture resolution information" includes parameters indicating the resolution of picture 12 (pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, etc.).

[0078] In addition, the picture parameter set includes "sub-picture identification information" configured by parameters related to the identification of sub-picture 13. For example, the "sub-picture identification information" includes the identification information of each sub-picture 13 included in picture 12 (pps_subpic_id[i]). For example, numbers can be sequentially assigned to each sub-picture 13 in picture 12 (e.g., assigning integer values that increase by "+1", such as 0, 1, 2...), and this number can be used as the identification information of sub-picture 13.

[0079] In addition, the picture parameter set includes tile information configured by parameters related to the configuration of tiles in picture 12. If tiles with the same width or height continue to the edge of the screen, this tile information can be omitted.

[0080] In addition, as Figure 4 shown, the picture parameter set includes slice information (slice information) configured by parameters related to the configuration of slices in the picture. In the case where all sub-pictures 13 are configured by one slice, this slice information can be omitted.

[0081] In the case of extracting some sub-pictures 13 from such a VVC bitstream 10 and decoding them, due to the change in the configuration of sub-pictures in the picture, the above information in the picture parameter set may change. That is, in this case, it is necessary to update the picture parameter set of the VVC bitstream 10 (picture 12) and generate a picture parameter set corresponding to the bitstream of the extracted sub-pictures.

[0082] <VVC File Format>

[0083] Meanwhile, as a method of storing a file of a picture encoded by VVC, for example, as described in Non-Patent Documents 2 to Non-Patent Document 4, a VVC file format using the International Organization for Standardization Base Media File Format (ISOBMFF) is being developed. This format is a file container specification for the international standard technology of the Video Compression Moving Picture Experts Group - 4 (MPEG-4).

[0084] Figure 5 is a diagram showing an example of the configuration of tracks in the VVC file format. As Figure 5As shown, VVC track 20 is a track used to store the VVC bitstream and includes a moov box and an mdat box. The moov box stores metadata related to the track, while the mdat box stores the VVC bitstream.

[0085] In the case of configuring multiple sub-pictures in an image, for example in this VVC file format, the sub-pictures in the image can be stored in multiple tracks, or they can be stored together in one track.

[0086] For example, when sub-images are divided into multiple tracks and stored, independent decoding of some sub-images can be achieved by selecting the track to be decoded during decoding. In this case, information about the merging of the selected tracks (metadata) is stored in a different track than the track storing the bitstream of the sub-images. This track is also called the VVC base track. Furthermore, the track storing the bitstream of the sub-images is also called the VVC sub-image track.

[0087] Figure 6 This example demonstrates the syntax of a VVC subpicture order entry (VvcSubpicOrderEntry) stored in a VVC base track when subpictures are divided into multiple tracks and stored. Figure 6 As shown, in the VVC base track, the VisualSampleGroupEntry is extended to generate a VVC subpic order entry (VvcSubpicOrderEntry() extends VisualSampleGroupEntry('spor')). The VVC subpic order entry stores information defining the subpic order sample group, which is the group of subpicks to be reconstructed. The subpic order sample group can include information such as the "number of subpicks," the "decoding order of the subpicks," and "subpic ID rewriting information." The "number of subpicks" indicates the number of subpicks to be merged, the "decoding order of the subpicks" indicates the decoding order of the subpicks to be merged, and the "subpic ID rewriting information" is information related to the rewriting of the subpicks' identification information. For example, subp_track_ref_idx indicates the track or track group that includes the subpicks to be merged. Furthermore, subpic_id_info_flag indicates whether "subpic ID rewriting information" is present or absent.

[0088] also, Figure 7 This example illustrates the syntax of a VVC subpick ID entry (VvcSubpicIDEntry) stored in a VVC subpick track when subpicks are divided into multiple tracks and stored. Figure 7As shown, in the VVC subpick track, the VisualSampleGroupEntry is extended to generate a VVC subpick ID entry (VvcSubpicIDEntry()extends VisualSampleGroupEntry('spid')). The VVC subpick ID entry stores information defining a subpick ID sample group, which is a group of subpick identifiers (subpic_id) included in the samples within the VVC subpick track. The subpick ID sample group indicates the identifiers (subpic_id) of the subpicks included in the track in the decoding order.

[0089] As mentioned above, sub-images within an image can be stored collectively in one track. Figure 5 In the example, VVC track 20 stores the bitstream of an image comprising six sub-images as sample 23 in an mdat box. That is, sample 23 comprises the bitstream 24 of six sub-images (sub-image 1 data to sub-image 6 data).

[0090] In this case, a tile region loop entry (Trif) 21 and an empty cell mapping entry (NALUMapEntry) 22 are formed in the moov box of VVC track 20.

[0091] Tile region loop entry 21 stores information defining a tile region sample group, which is a sample group indicating the tile regions of the image. Empty cell mapping entry 22 stores information defining an empty cell mapping sample group (NAL cell mapping sample group), which is a sample group used to classify the empty cells (NAL cells) included in the sample 23 of each tile region.

[0092] In the tile region cycle entry 21, the groupID is stored as an identifier. That is, the groupID is the identification information for the tile region. For example, in... Figure 8 As shown in Figure A, when four tile regions are formed in the image, different groupIDs are assigned to each region (groupID = 1 to groupID = 4). Tile region cycle entry 21 also stores information specifying the tile region (position and size). In VVC, a tile region can contain one or more sub-images. That is, as shown in Figure A... Figure 8 As shown in B, a tile region can be configured by one sub-image or by multiple sub-images.

[0093] Figure 9 This shows an example of the syntax for the tile region loop entry 21. For example... Figure 9As shown, the tile region loop entry 21 stores the identifier (groupID) of the tile region loop entry 21, parameters indicating the position of the tile region (horizontal_offset, vertical_offset), parameters indicating the size of the tile region (region_width, region_height), parameters indicating the dependencies between tile regions (dependency_tile_count, dependencyTileGroupID), etc.

[0094] Empty cell mapping entry 22 stores information indicating the correspondence between empty cells and groupIDs. Since the groupID is assigned to the tile region as described above, empty cells and... Figure 10 Any groupID shown in A corresponds to this. The correspondence between empty cells and groupIDs is indicated in empty cell mapping entry 22. Figure 10 Example of the syntax for empty cell mapping entry 22 is shown in section B. Figure 10 As shown, information (NALU_start_number) indicating the empty cell corresponding to each groupID is displayed.

[0095] <Use Case: Omnidirectional Video>

[0096] Figure 11 This is a diagram illustrating use cases for applying sub-images. For example, such as... Figure 11 As shown in A, the video on each side of an omnidirectional video content (cube map) comprising video in six directions—up, down, left, right, forward, and backward—can be used as a sub-image. In the case of such omnidirectional video content, various scenarios exist, such as the case where only one of the six surfaces is decoded, the case where only two surfaces are decoded, and so on, as well as the case where all six surfaces are decoded.

[0097] Therefore, as Figure 11 As shown in B, the video of all surfaces is stored in a VVC track, and some subsamples can be extracted appropriately to handle each of the above cases.

[0098] <Example: V3C Bitstream>

[0099] Furthermore, there exists video-based point cloud compression (V-PCC), in which a point cloud, which is a set of points possessing both positional and attribute information (color, reflection, etc.) in three-dimensional space, is segmented to form regions. Attribute images (color information, etc.) and geometric images (composed of depth information) are generated by planar projection of the point cloud into each region. Eye diagram images and atlas information (information used to reconstruct the point cloud from image blocks) are also generated, and the three images are encoded by a motion picture codec. Non-Patent Document 5 discloses a technique for storing V3C bitstreams in ISOBMFF, aiming to improve the efficiency of reproduction processing and network distribution from local storage of bitstreams encoded by such V-PCC (referred to as V3C bitstreams).

[0100] Video-based point cloud data (V3C) file format is a file format that uses ISO basic media files to store V-PCC. The V3C file format can store point cloud data in multiple tracks (V3C track, occ, geo, att), for example, as... Figure 12 As shown in A. In this case, the metadata used to reconstruct the point cloud is included in the V3C track. Furthermore, the V3C track is a track (video component track) that includes the occupancy map, geometry, and attributes of each of the V-PCC.

[0101] Furthermore, the V3C file format can separate tracks for each section, for example, such as... Figure 12 As shown in B, this allows for partial access to point cloud data (independent decoding of specific regions of the point cloud). In this case, the following track configuration is possible. A V3C track is a V3C atlas tile track for a specific region. Each V3C atlas tile track is a corresponding video component track containing the occupancy map, geometry, and attributes of its atlas tiles. For example, each video component track can store and independently decode a VVC bitstream. In the case of a VVC bitstream, the VVC file format is applied to each video component track.

[0102] Even with content in this V3C file format, to achieve partial access, one could consider setting each partial region as a sub-image, storing all partial regions in a single VVC track, and making it possible to extract some sub-images appropriately.

[0103] <Parameter Set Reconstruction>

[0104] As mentioned above, in the VVC file format, multiple sub-pictures can be stored in a single track, and some sub-pictures can be extracted and decoded. However, in this case, a bitstream is reconstructed using the extracted sub-pictures, and then decoded. Since the configuration of the sub-pictures differs between the bitstream before and after reconstruction, the parameter set (e.g., sequence parameter set or picture parameter set) cannot be used as is. Therefore, it is necessary, for example, to derive the parameter set corresponding to the reconstructed bitstream by updating the parameter set before reconstruction.

[0105] In other words, complex tasks are required, such as analyzing the bitstream and its parameter set before reconstruction, updating the parameter set, and deriving the parameter set corresponding to the reconstructed bitstream. This can increase the load on the reconstruction process. Note that the reconstruction process includes at least the decoding of the bitstream of the contents stored in the file and decoding-related processing (e.g., reconstructing the bitstream to be decoded). Furthermore, the reconstruction process may include processing to generate a display image using the data obtained through decoding, or it may include other processing.

[0106] For example, when a content file with multiple sub-images is distributed from a server, and the client device generates a display image by decoding only some of the sub-images stored in that file, there is a possibility of increased load on the client device. Typically, the processing power of a client device is lower than that of the server. Therefore, due to such increased load, there is a possibility of processing delays or failures. In other words, to prevent processing delays or failures, the processing power of the client device needs to be increased, which may increase costs.

[0107] Note that, as described above, in the case of storing each sub-image across multiple tracks, the set of parameters required for reconstructing the bitstream using the VVC base track can be stored. However, in the VVC base track, it is not possible to extract the desired sub-image from multiple sub-images existing within the track.

[0108] <2. Extracting the basic track>

[0109] Method 1

[0110] Therefore, as Figure 13 As shown in the top row of the table, VVC extract base tracks with reconstruction information are provided and stored in a file (Method 1). This reconstruction information includes, for example, the set of parameters required to reconstruct the bitstream from some sub-images. Then, in the case of reproducing some sub-images, a reconstructed bitstream is generated based on the information stored in the VVC extract base tracks.

[0111] Figure 14 This is a diagram illustrating a primary configuration example of a document applying this technology. Figure 14 The file 100 shown is a VVC file format that conforms to the ISOBMFF standard for storing VVC bitstreams. For example... Figure 14 As shown, file 100 has a VVC bitstream track 110 and a VVC extraction base track 120.

[0112] VVC bitstream track 110 is a track for storing VVC bitstreams, similar to VVC track 20 described above. VVC bitstream track 110 can store VVC bitstreams with multiple sub-pictures. In this example, within the mdat box of VVC bitstream track 110, a bitstream of a picture comprising six sub-pictures is stored as sample 115. That is, sample 115 comprises a bitstream 116 of six sub-pictures (sub-picture 1 data to sub-picture 6 data).

[0113] Furthermore, the tile region loop entry (Trif) 111 and the empty cell mapping entry (NALUMapEntry) 112 are stored in the moov box of the VVC bitstream track 110. Similar to the tile region loop entry 21 described above, information defining the tile region sample group is stored in the tile region loop entry 111. Similar to the empty cell mapping entry 22 described above, the empty cell mapping entry 112 stores information indicating the correspondence between empty cells and groupIDs.

[0114] In addition, sample entry (SampleEntry(vvc1)) 113 is stored in the moov box of VVC bitstream track 110. Sample entry 113 stores VVC configuration box (VvcConfigurationBox) 114.

[0115] VVC Extraction Base Track 120 stores reconstruction information required to reconstruct the bitstream from some sub-pictures. The reproduction device can extract the bitstream 116 of the desired sub-pictures from VVC Bitstream Track 110 based on the reconstruction information stored in VVC Extraction Base Track 120, and reconstruct the bitstream of the extracted sub-pictures.

[0116] For example, the reproduction device first determines the region to be reproduced based on information stored in the tile region loop entry 111 of the VVC bitstream track 110. Therefore, a groupID indicating the region to be reproduced is determined. Next, the reproduction device searches for the VVC extraction base track 120 that includes the region to be reproduced. Then, the reproduction device updates (rewrites) the parameter sets (such as the sequence parameter set and the picture parameter set) based on information stored in the searched VVC extraction base track 120. Then, the reproduction device uses the empty cell mapping sample group defined in the empty cell mapping entry 112 of the VVC bitstream track 110 to extract the empty cells of the region to be reproduced from the sample 115, and reconstructs the sample using the information stored in the VVC extraction base track 120.

[0117] As described above, the reconstructing device can easily reconstruct the bitstream without analyzing the VVC bitstream stored in the VVC bitstream track 110. Therefore, the increase in the load of the reconstructing process can be suppressed.

[0118] Note that in Figure 14 In the example case, file 100 includes VVC extraction base tracks 121 (for reconstructing one sub-image (1 side)), 122 (for reconstructing two sub-images (2 sides), 123 (for reconstructing three sub-images (3 sides), 124 (for reconstructing four sub-images (4 sides), and 125 (for reconstructing five sub-images (5 sides)). In this way, VVC extraction base tracks can be generated for each number of sub-images to be reconstructed.

[0119] <Method 1-1>

[0120] Note that, as Figure 13 As shown in the second row at the top of the table, the track used for reproduction and the extraction information stored in the track, specifying the sub-images to be reproduced, can be stored in the VVC extraction base track (Method 1-1). Then, in the case of reproducing some sub-images, a reconstructed bitstream can be generated based on the extraction information stored in the VVC extraction base track.

[0121] For example, VVC extraction base track 120 can store sub-picture specification information as reconstruction information, which specifies the information to be extracted from the sub-picture in VVC bitstream track 110.

[0122] For example, in an information processing method, files are generated to store bitstream tracks and extract basic tracks. The bitstream tracks store bitstreams containing sub-images within frames, and the basic tracks store sub-image specified information, which is used to specify the sub-images to be extracted from the bitstream tracks.

[0123] For example, the information processing apparatus includes: a file generation unit that generates and stores a bitstream track and an extraction base track, the bitstream track storing a bitstream containing sub-image content within a frame, and the extraction base track storing sub-image specification information, the sub-image specification information being used to specify the sub-image to be extracted from the bitstream track.

[0124] By doing so, a file containing extracted base tracks that store information specifying the sub-pictures is generated. Therefore, the reproduction device can extract the desired sub-pictures from the bitstream tracks based on the extracted base tracks and reconstruct the bitstream. In other words, the reproduction device can easily reconstruct the bitstream without analyzing it. Therefore, the increase in the load on the reproduction process can be suppressed.

[0125] Furthermore, for example, in an information processing method, based on the file of the stored bitstream track and the extracted base track, sub-images are extracted from the bitstream track and a bitstream is generated. The bitstream track stores the bitstream containing the content of the sub-images within the frame. The extracted base track stores the sub-image specification information, which is used to specify the sub-images to be extracted from the bitstream track.

[0126] For example, the information processing apparatus includes: a bitstream generation unit that extracts a basic track from a file that stores bitstream tracks and extracts basic tracks, extracts sub-images from the bitstream tracks, and generates a bitstream; the bitstream tracks store bitstreams containing sub-image content within frames; and the basic tracks store sub-image specification information, which is information used to specify the sub-images to be extracted from the bitstream tracks.

[0127] By doing so, desired sub-pictures can be extracted from the bitstream track based on the extraction base track, including sub-picture specification information, and the bitstream can be reconstructed. In other words, the bitstream can be easily reconstructed without analyzing it. Therefore, the increase in the load of the reproduction processing can be suppressed.

[0128] Note that the sub-image specification information may include tile region identification information, which is the identification information of the tile region that includes the sub-image. For example, see reference. Figure 8 The described groupID can be used as sub-image specification information. By doing so, the reproduction device can more easily specify the tile area (sub-image) to be reproduced.

[0129] <Method 1-1-1>

[0130] Note that, as Figure 13 As shown in the third row at the top of the table, sample groups can be defined, and extraction information that allows the client to select one from multiple sub-image sets can be stored in the sample groups (Method 1-1-1). Then, in the case of reproducing some sub-images, the set of sub-images to be extracted can be selected based on the sample groups.

[0131] For example, a VisualSampleGroupEntry(spex) can be extended to generate a VVC SubpictureExtractionEntry(), and information about the subpicture extraction sample group, which defines the sample group of the subpicture to be extracted (reproduced), can be stored in the VVC SubpictureExtractionEntry.

[0132] The subpicture extraction sample set may include, for example, information such as groupID (also called region set) indicating the loop entries of tile regions for candidate tracks of the subpicture to be extracted. Note that when directly specifying tile regions, only the region set including one tile region needs to be stored. Furthermore, the subpicture extraction sample set may include information related to the rewriting (updating) of subpicture identifier information (subpic_id rewriting information) as identification information for the subpicture. As described above, by applying region sets, the reproduction device can more easily extract multiple tile regions.

[0133] The reproduction device (client) selects and extracts a groupID from the region set of the sample group extracted from the sub-image.

[0134] Figure 15 This example demonstrates the syntax for the VVC subpicture extraction entry (VvcSubpictureExtractionEntry()) in this scenario. Figure 15 In the example case, the following parameters can be stored in the VVC sub-image extraction entry. `num_region_set` indicates the number of region sets. `num_region_in_set` indicates the number of tile regions included in the region set. `groupID` indicates the group ID of the tile regions included in the region set. `num_regions` indicates the number of tile regions to extract. `region_idx` is specified using either `region_set_id` or `groupID` as identifier information for the region set.

[0135] For example, extracting a base track can be set up as a set of sub-images extracted as a bitstream, and specific information for each sub-image can be stored for each set.

[0136] In addition, the extracted basic orbitals can also store the set of identification information as a set of identification information.

[0137] By setting the aforementioned region set, extraction information for multiple modes can be stored in a single VVC extraction base track. In other words, multiple combinations of sub-images to be extracted can be defined within a single VVC extraction base track. Therefore, the playback device can extract various combinations of more sub-images based on such a VVC extraction base track. That is, the number of extraction modes (combinations) for sub-images can be increased without increasing the number of tracks. Therefore, redundant tracks can be collected, and the reduction in coding efficiency can be suppressed.

[0138] Note that in Figure 15 In the syntax example shown, the information following the line "if(subpic_id_info_flag){" is about rewriting the subpicture identification information. That is, information used to update parameter sets such as sequence parameter sets and picture parameter sets to correspond to the reconstructed bitstream is stored in the VVC subpicture extraction entry. More specifically, information used to update the subpicture identification information included in the parameter sets to correspond to the reconstructed bitstream is stored in the VVC subpicture extraction entry.

[0139] In other words, extracting the basic track can also store parameter set update information, which is used to update the parameter set to correspond to the bitstream of the extracted sub-image.

[0140] For example, parameter set update information may include sub-image update information, which is used to update the sub-image identifier information, which is the identifier information of the sub-image, to correspond with the bitstream of the extracted sub-image.

[0141] By doing so, the reconstruction device can more easily map the parameter set to the reconstructed bitstream based on the information stored in the extracted base track. For example, the reconstruction device can more easily map the sub-image identification information included in the parameters to the reconstructed bitstream based on the information stored in the extracted base track.

[0142] <3. Expansion of Sub-image ID Sample Groups>

[0143] <Method 1-2>

[0144] like Figure 13 As shown in the fourth row at the top of the table, the sub-image ID sample group can be expanded, and mapping information indicating the correspondence between sub-images and tile regions can be stored (Methods 1-2). For example, as such mapping information, information indicating the tile region including the sub-image can be stored in the sub-image ID sample group.

[0145] For example, in Figure 7 In the case of the sub-image ID sample group shown, the relationship between the sub-image and the tile region (which includes the sub-image) is unknown. (See reference...) Figure 8 As mentioned above, since tile regions and sub-images do not always correspond one-to-one, there may be situations where they cannot be based on... Figure 15 The example shown illustrates the case where a sample group of sub-images is extracted to specify the desired sub-image for the desired tile region (groupID).

[0146] Therefore, in the subpic ID sample group, the tile region (groupID) that includes the subpic (subpic_id) is indicated.

[0147] Figure 16 This example demonstrates the syntax of `VvcSubpicIDEntry()`, which stores information defining a sample group of subpicture IDs in this context. Figure 16 As shown, the tile region identifier (groupID) adjacent to the subpic ID (subpic_id) and including the subpic ID corresponding to the subpic ID (subpic_id) is stored in VvcSubpicIDEntry(). Therefore, the correspondence between the subpic (subpic_id) and the tile region (groupID) is clearly indicated.

[0148] In other words, for each sub-image stored in the bitstream track, the bitstream track can also store sub-image identification information and tile region identification information including the tile region of the sub-image.

[0149] By doing so, the reproduction device can more easily specify the subpicks (subpic_id) to be included in the desired tile region based on the information stored in the subpick ID sample group. Therefore, the reproduction device can more easily rewrite (update) the subpick identification information (subpic_id) based on the information extracted from the subpick sample group.

[0150] <Method 1-2-1>

[0151] like Figure 13 As shown in the fifth row at the top of the table, the storage of mapping information indicating the correspondence between the aforementioned sub-images and tile regions can be omitted (Method 1-2-1).

[0152] Figure 17 This example demonstrates the syntax of `VvcSubpicIDEntry()`, which stores information defining a sample group of subpicture IDs in this context. Figure 17As shown, in this case, `all_subpic_id_is_same_asgroupID_flag` is stored in `VvcSubpicIDEntry()`. `all_subpic_id_is_same_asgroupID_flag` is a flag indicating whether all subpicks (subpic_id) in the image match (one-to-one) with the tile region (groupID). When `all_subpic_id_is_same_asgroupID_flag` is true (e.g., 1), it indicates that all subpicks (subpic_id) in the image match the tile region (groupID). Conversely, when `all_subpic_id_is_same_asgroupID_flag` is false (e.g., 0), it indicates that at least some subpicks do not have a one-to-one correspondence with the tile region (a tile region contains multiple subpicks). Therefore, `groupID` is only stored when `all_subpic_id_is_same_asgroupID_flag` is false (e.g., 0). In other words, if all subpics (subpic_id) in an image match (correspond one-to-one) with the tile region (groupID), the storage of groupID is omitted.

[0153] In other words, the bitstream track can store sub-image identification information only for sub-images whose sub-image identification information matches the tile region identification information.

[0154] When all subpicks (subpic_id) and tile regions (groupID) in an image have a one-to-one correspondence, even without mapping information, groupID and subpic_id can be easily associated one-to-one. Therefore, by doing so, unnecessary increases in code volume can be suppressed, thus preventing a decrease in coding efficiency.

[0155] <4. Combination of VVC Basic Tracks and VVC Additional Basic Tracks>

[0156] Method 2

[0157] like Figure 13 As shown in the sixth row at the top of the table, sub-images can be extracted from multiple tracks using VVC base tracks and VVC extract base tracks (Method 2). That is, VVC extract base tracks only supports extraction from one track, and VVC base tracks can be used in combination when performing extraction from multiple tracks.

[0158] For example, such as Figure 18As shown in A, some sub-images are extracted from two images, and the extracted sub-images are merged into one image. In this case, as... Figure 18 As shown in B, when extracting sub-images from an image, VVC can be applied to extract the information of the basic track, and when merging the extracted sub-images, the information of the basic track can be applied.

[0159] In other words, the file storing bitstream tracks and extracting base tracks may also include merging base tracks, which stores information about the merging of multiple sub-images extracted from different bitstream tracks based on the extracted base tracks.

[0160] Any VVC extraction base track described in <2. Extracting the base track> and <3. Expanding the sub-image ID sample group> can be applied to this VVC extraction base track.

[0161] also, Figure 19 The example shown is an example of the syntax for the VVC Subpic Order Entry ((VvcSubpicOrderEntry()extends VisualSampleGroupEntry('spor')) of the VVC basic track.

[0162] like Figure 19 As shown, in this example, `extract_track_exist_flag` is stored in the VVC sub-image sequence entry. `extract_track_exist_flag` is a flag indicating whether the base track extracted by VVC is included in the track to be referenced. For example, if `extract_track_exist_flag` is true (e.g., 1), it indicates that the base track extracted by VVC is included in the track to be referenced. Conversely, if `extract_track_exist_flag` is false (e.g., 0), it indicates that the base track extracted by VVC is not included in the track to be referenced.

[0163] By storing such an `extract_track_exist_flag`, it is easy to identify whether the VVC extraction base track is included in the track to be referenced. That is, the VVC extraction base track and the VVC base track, as described above, can be used more easily together. Furthermore, the VVC extraction base track can be included in tracks that can be merged into the VVC base track. Additionally, it is easy to determine whether the VVC extraction base track is included in the VVC base track.

[0164] <5. Application of V3C File Format>

[0165] Method 3

[0166] like Figure 13 As shown in the bottom row of the table, this technique can be applied to the V3C file format (Method 3).

[0167] For reference Figure 12 As described above, V-PCC bitstreams can be stored in ISOBMFF files using the V3C file format. This technique can be applied to such V3C file formats, and partial access can be achieved using sub-images.

[0168] In other words, the content can be a point cloud that represents a set of points as three-dimensional objects, and sub-images can be configured from independently decodeable portions of the point cloud.

[0169] For example, such as Figure 20 As shown in A, partial access to the V-PCC bitstream can be achieved by using VVC to extract the basic track. Figure 20 In the example shown in A, there exists a V3C atlas tile track. In such a case, as... Figure 20 As shown in the dashed box in A, only the VVC extraction base track needs to be provided for each of the occupied map, geometry, and attributes.

[0170] The VVC extraction base track for each example described in any of the following items—<2. Extracting the Base Track>, <3. Expanding the Sub-Image ID Sample Group>, and <4. Combination of VVC Base Track and VVC Additional Base Track>—can be applied to that VVC extraction base track. Furthermore, multiple of these examples can be combined. That is, the extraction base track storing sub-image specification information is stored in a file; this sub-image specification information is used to specify the sub-images to be extracted.

[0171] During reproduction, it is only necessary to use VVC to extract the basic track for each of the occupied map, geometry, and attributes to extract and decode the necessary sub-images.

[0172] In addition, such as Figure 20 As shown in B, the V3C file format can be a multi-track configuration, where each of the occupancy map, geometry, and attributes is stored in a different track from each other, where there is no V3C atlas track.

[0173] However, the track for storing the occupied map is a VVC track that includes multiple sub-images. The same applies to geometry and attributes. In the case of performing partial reproduction, this can be achieved by extracting and reproducing specific sub-images.

[0174] <6. First Implementation Method>

[0175] <Document generation device>

[0176] Each of the above-described techniques can be applied to any device. Figure 21 This is a block diagram illustrating an example configuration of a document generation device as an aspect of an information processing apparatus applying this technology. Figure 21 The file generation device 300 shown is a device that stores a VVC bitstream obtained by encoding video content, including sub-pictures, using ISOBMFF.

[0177] The file generation device 300 applies the aforementioned technology to store multiple sub-images in one track and stores a VVC bitstream in the file, enabling the independent reproduction of some sub-images. In other words, the file generation device 300 stores a bitstream track and an extraction base track. The bitstream track stores the bitstream containing the content of sub-images within a frame, while the extraction base track stores sub-image specification information, which is used to specify the sub-images to be extracted from the bitstream track in the file.

[0178] Note that in Figure 21 The diagram shows the main processing units, data flow, etc., and Figure 21 The examples shown are not necessarily all. That is to say, the file generation device 300 may contain... Figure 21 The processing unit not shown in the diagram is a block, or it may exist. Figure 21 The arrows or other symbols not shown in the text represent processing or data flow.

[0179] like Figure 21 As shown, the file generation device 300 includes a control unit 301 and a file generation processing unit 302. The control unit 301 controls the file generation processing unit 302. The file generation processing unit 302 is controlled by the control unit 301 and performs processes related to file generation. For example, the file generation processing unit 302 acquires data containing sub-images in an image, encodes the data, and generates a VVC bitstream. The file generation processing unit 302 also stores the generated VVC bitstream in an ISOBMFF file and outputs the file to the outside of the file generation device 300.

[0180] The file generation and processing unit 302 includes an input unit 311, a preprocessing unit 312, an encoding unit 313, a file generation unit 314, a recording unit 315, and an output unit 316.

[0181] Input unit 311 acquires data containing sub-images from an image and provides this data to preprocessing unit 312. Preprocessing unit 312 extracts information required for file generation from the content data. Preprocessing unit 312 provides the extracted information to file generation unit 314. Furthermore, preprocessing unit 312 provides the content data to encoding unit 313.

[0182] Encoding unit 313 uses the VVC method to encode the data provided from preprocessing unit 312 to generate a VVC bitstream. Encoding unit 313 provides the generated VVC bitstream to file generation unit 314.

[0183] The file generation unit 314 stores the VVC bitstream provided by the encoding unit 313 into an ISOBMFF file. At this time, the file generation unit 314 appropriately stores the information provided by the preprocessing unit 312 into the file.

[0184] Furthermore, the file generation unit 314 applies the technology described in any one of <2. Extracting the basic track>, <3. Expanding the sub-image ID sample group>, <4. Combination of VVC basic track and VVC additional basic track>, and <5. Application of V3C file format> or combinations thereof, and generates a file storing the VVC bitstream.

[0185] In other words, the file generation unit 314 generates files for storing bitstream tracks and extracting basic tracks. The bitstream track stores the bitstream containing sub-image content within the frame, and the extracting basic track stores sub-image specified information. The sub-image specified information is used to specify the sub-image to be extracted from the bitstream track.

[0186] The file generation unit 314 provides the generated file to the recording unit 315. The recording unit 315 includes any recording medium such as a hard disk or semiconductor memory, and records the file provided from the file generation unit 314 in the recording medium. In addition, the recording unit 315 reads the file recorded in the recording medium according to a request from the control unit 301 or the output unit 316 or at a predetermined time, and provides the file to the output unit 316.

[0187] The output unit 316 acquires the file provided by the recording unit 315 and outputs the file to an external part of the file generation device 300 (e.g., a distribution server, a reproduction device, etc.).

[0188] With this configuration, the file generation device 300 can store the reconstruction information required to reconstruct the bitstream from some sub-images in the extraction base track. Therefore, the reproduction device for reproducing the file can extract the bitstream of the desired sub-images from the bitstream track based on the reconstruction information stored in the extraction base track, and reconstruct the bitstream of the extracted sub-images. Thus, the increase in the load of the reproduction process can be suppressed.

[0189] <File Generation and Processing Flow>

[0190] Reference Figure 22 The flowchart in the middle describes the process by Figure 21 An example of the file generation process executed by the file generation unit 314.

[0191] When the file generation process begins, in step S301, the file generation unit 314 of the file generation device 300 generates a VVC bitstream track that stores the VVC bitstream.

[0192] In step S302, the file generation unit 314 determines the combination of sub-images to be extracted.

[0193] In step S303, the file generation unit 314 generates reconstruction information required to reconstruct the bitstream from some sub-images for each combination of sub-images determined in step S302. That is, the file generation unit 314 generates sub-image specification information as reconstruction information, which is used to specify the sub-images to be extracted in the bitstream track.

[0194] In step S304, the file generation unit 314 stores the reconstruction information generated in step S303 in the VVC extraction base track and generates a file. That is, the file generation unit 314 generates a file that stores the bitstream track and the extraction base track. The bitstream track stores the bitstream containing sub-images within the frame, and the extraction base track stores sub-image specification information, which is used to specify the sub-images to be extracted from the bitstream track.

[0195] When the file is generated, the file generation process ends.

[0196] As described above, by performing each process, the file generation device 300 can generate an extraction base track storing the reconstruction information required to reconstruct the bitstream from some sub-images, and generate a file storing the extraction base track. Therefore, the reproduction device for reproducing the file can extract the bitstream of the desired sub-images from the bitstream track based on the reconstruction information stored in the extraction base track, and reconstruct the bitstream of the extracted sub-images. Thus, the increase in the load of the reproduction process can be suppressed.

[0197] <7. Second Implementation>

[0198] <Client Device>

[0199] Figure 23 This is a block diagram illustrating an example configuration of a reproduction device as an aspect of an information processing apparatus to which this technology is applied. Figure 23The client device 400 shown is a reproduction device that decodes the VVC bitstream stored in a VVC file format file and generates and displays a display image of the generated motion picture content. For example, the client device 400 decodes the VVC bitstream stored in a file generated by the file generation device 300 and generates and displays a display image of the generated motion picture content. At this time, the client device 400 applies the above-described technology to extract some sub-images from the file, reconstruct the bitstream, and decode the reconstructed bitstream.

[0200] Note that in Figure 23 The diagram shows the main processing units, data flow, etc., and Figure 23 The examples shown are not necessarily all of them. That is to say, client device 400 may contain... Figure 23 The processing unit not shown in the diagram is a block, or it may exist. Figure 23 The arrows or other symbols not shown in the text represent processing or data flow.

[0201] like Figure 23 As shown, the client device 400 has a control unit 401 and a playback processing unit 402. The control unit 401 performs processing related to the control of the playback processing unit 402. The playback processing unit 402 performs processing related to the playback of video content stored in a file. For example, the playback processing unit 402 is controlled by the control unit 401 and obtains files from a distribution server (not shown), etc.

[0202] This file conforms to the VVC file format generated by applying this technology. For example, the file is generated by file generation device 300. That is, the file is used to store bitstream tracks and extract basic tracks. The bitstream tracks store the bitstream containing sub-pictures within a frame, and the extract basic tracks store sub-picture specification information, which specifies the sub-pictures to be extracted from the bitstream tracks.

[0203] The playback processing unit 402 performs playback processing on the acquired file, decodes the bitstream of the video content stored in the file, and generates and displays the display image of the video content.

[0204] The playback processing unit 402 includes a file acquisition unit 411, a file processing unit 412, a decoding unit 413, a display information generation unit 414, a display unit 415, a measurement unit 416, and a display control unit 417.

[0205] The file acquisition unit 411 acquires a VVC file format provided from an external source (e.g., a distribution server, file generation device 300, etc.) of the client device 400. As described above, this file stores a bitstream track and an extraction base track. The bitstream track stores a bitstream containing sub-pictures within frames, and the extraction base track stores sub-picture specification information, which specifies the sub-pictures to be extracted from the bitstream track. The file acquisition unit 411 provides the acquired file to the file processing unit 412.

[0206] The file processing unit 412 acquires files provided by the file acquisition unit 411. The file processing unit 412 acquires measurement results provided by the measurement unit 416. The file processing unit 412 acquires control information provided by the display control unit 417.

[0207] Using this information, file processing unit 412 extracts some sub-images from the file and generates a bitstream of the extracted sub-images. At this time, file processing unit 314 applies the technique described in any one of <2. Extracting basic tracks>, <3. Expansion of sub-image ID sample groups>, <4. Combination of VVC basic tracks and VVC additional basic tracks>, and <5. Application of V3C file format> or combinations thereof, and performs processing on the file.

[0208] In other words, the file processing unit 412 extracts sub-images from the bitstream tracks stored in the file based on the extraction base tracks stored in the file, and generates a bitstream of the extracted sub-images.

[0209] The file processing unit 412 provides the generated bitstream to the decoding unit 413. The decoding unit 413 decodes the provided bitstream and generates video content data of the extracted sub-images. The decoding unit 413 provides the generated video content data to the display information generation unit 414.

[0210] The display information generation unit 414 acquires video content data provided by the decoding unit 413. Furthermore, the display information generation unit 414 acquires control information provided by the display control unit 417. Then, the display information generation unit 414 generates a display image, etc., from the acquired video content data based on the control information. The display information generation unit 414 provides the generated display image, etc., to the display unit 415. The display unit 415 includes a display device and uses the display device to display the provided display image. The measurement unit 416 measures arbitrary information and provides the measurement results to the file processing unit 412. The display control unit 417 controls the display by providing control information to the file processing unit 412 and the display information generation unit 414.

[0211] Using this configuration, the client device 400 can extract a portion of the bitstream of the sub-image as a file based on the extraction base track containing the reconstruction information required to reconstruct the bitstream from a portion of the sub-image. Therefore, the client device 400 can more easily reconstruct the extracted bitstream of the sub-image. This reduces the increase in the load of the reproduction processing.

[0212] <Reproduction Process>

[0213] Reference Figure 24 The flowchart description in the document is provided by Figure 23 An example of the reproduction process executed by the client device 400.

[0214] When the reproduction process begins, in step S401, the file processing unit 412 of the client device 400 acquires a file in VVC file format.

[0215] In step S402, the file processing unit 412 determines some tile regions to be reproduced based on the information stored in the VVC bitstream track, which is stored in the file obtained in step S401.

[0216] In step S403, the file processing unit 412 searches for and obtains the VVC extraction base track from the file, which includes the tile region determined in step S402.

[0217] In step S404, the file processing unit 412 extracts the reconstruction information of the basic track from the VVC obtained through the processing in step S403, extracts sub-images corresponding to some tile regions to be reproduced from the file, and reconstructs their bitstream.

[0218] In other words, the file processing unit 412 extracts the desired sub-image from the bitstream track of the bitstream containing the content of the sub-image stored in the file and storing the sub-image specification information based on the extraction base track stored in the file and storing the sub-image specification information. The sub-image specification information is used to specify the sub-image to be extracted.

[0219] In step S405, the decoding unit 413 decodes the bitstream reconstructed in step S404 and obtains the video image content data of the desired tile area. The display information generation unit 414 generates a display image. The display unit 415 displays the display image.

[0220] When the processing in step S405 is completed, the reproduction process ends.

[0221] As described above, by performing each process, the client device 400 can extract the desired sub-pictures from the bitstream track based on the extraction base track, which includes sub-picture specification information, and reconstruct the bitstream. That is, it can easily reconstruct the bitstream without analyzing it. Therefore, the increase in the load of the reproduction processing can be suppressed.

[0222] <8. Appendix>

[0223] Computer

[0224] The above series of processes can be performed by hardware or software. In the case where the series of processes are performed by software, the program constituting the software is installed in the computer. Here, the computer includes, for example, a computer integrated into dedicated hardware, a general-purpose personal computer capable of performing various functions by installing various programs, etc.

[0225] Figure 25 This is a block diagram illustrating an example configuration of computer hardware that performs the above series of processes through a program.

[0226] exist Figure 25 In the computer 900 shown, the central processing unit (CPU) 901, read-only memory (ROM) 902 and random access memory (RAM) 903 are interconnected via bus 904.

[0227] The input / output interface 910 is also connected to the bus 904. The input unit 911, output unit 912, storage unit 913, communication unit 914, and driver 915 are connected to the input / output interface 910.

[0228] Input unit 911 includes, for example, a keyboard, mouse, microphone, touchpad, input terminal, etc. Output unit 912 includes, for example, a display, speaker, output terminal, etc. Storage unit 913 includes, for example, a hard disk, RAM disk, non-volatile memory, etc. Communication unit 914 includes, for example, a network interface. Driver 915 drives removable media 921 such as a hard disk, optical disk, magneto-optical disk, or semiconductor memory.

[0229] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executes the program, thereby performing the series of processes described above. The RAM 903 also appropriately stores data, etc., required by the CPU 901 to perform various processes.

[0230] For example, a program executed by a computer can be applied by being recorded in a removable medium 921, such as a packaging medium. In this case, by attaching the removable medium 921 to the driver 915, the program can be installed in the storage unit 913 via the input / output interface 910.

[0231] Furthermore, the program can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0232] Alternatively, the program can be pre-installed in ROM 902 or storage unit 913.

[0233] <Applicable to this technology>

[0234] This technology can be applied to any image encoding / decoding method.

[0235] Furthermore, this technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0236] Furthermore, this technology can also be implemented as a partial configuration of a device, such as a processor (e.g., a video processor) as a system-wide integrated system (LSI), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a collection obtained by adding other functions to the unit (e.g., a video collection).

[0237] Furthermore, this technology can also be applied to network systems comprising multiple devices. For example, it can be implemented as cloud computing, where multiple devices collaborate, share, and process data via a network. For instance, it can be implemented in cloud services that provide image (moving image) related services to any terminal such as a computer, audiovisual (AV) device, portable information processing terminal, or Internet of Things (IoT) device.

[0238] Note that in this specification, "system" refers to a combination of multiple configuration elements (devices, modules (components), etc.), and it is irrelevant whether all components are housed in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a device in which multiple modules are housed in one enclosure, are both systems.

[0239] <Fields and Applications for which this technology is applicable>

[0240] Systems, devices, and processing units utilizing this technology can be used in any field, such as transportation, healthcare, crime prevention, agriculture, animal husbandry, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are also arbitrary.

[0241] For example, this technology can be applied to systems or devices provided for providing viewing content, etc. Furthermore, for example, it can be applied to systems and devices provided for transportation, such as traffic condition monitoring and automatic driving control. Furthermore, for example, this technology can be applied to systems or devices provided for safety purposes. Furthermore, for example, this technology can be applied to systems or devices for the automatic control of machines, etc. Furthermore, for example, this technology can be applied to systems and devices used in agriculture and animal husbandry. Furthermore, this technology can be applied to systems and devices for monitoring natural conditions such as volcanoes, forests and oceans, and wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0242] <8. Other>

[0243] Note that in this specification, "flag" refers to information used to identify multiple states, and includes not only information for identifying the two states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the value of "flag" can be, for example, binary 1 / 0 or ternary or more. That is, the number of bits constituting the "flag" is arbitrary and can be one or more bits. Furthermore, since it is assumed that the identification information (including flags) includes not only the identification information in the bitstream, but also the difference information of the identification information in the bitstream relative to a specific reference information, in this specification, "flag" and "identification information" include not only information, but also the difference information relative to the reference information.

[0244] Furthermore, various types of information (metadata, etc.) related to the encoded data (bitstream) can be sent or recorded in any form, as long as the information is associated with the encoded data. Here, the term "associated" means, for example, that one piece of data can be used (linked) while processing another piece of data. That is, data that are associated with each other can be collected as one piece of data or can be separate pieces of data. For example, information associated with the encoded data (image) can be transmitted on a transmission path different from the transmission path of the encoded data (image). Furthermore, for example, information associated with the encoded data (image) can be recorded on a different recording medium (or another recording area of ​​the same recording medium) than the encoded data (image). Note that this "association" may be a part of the data rather than the entire data. For example, an image and the information corresponding to that image can be associated with each other in any unit such as multiple frames, a single frame, or a portion of a frame.

[0245] Note that in this specification, terms such as “combine,” “reuse,” “add,” “integrate,” “include,” “store,” “put in,” “enter,” and “insert” refer to combining multiple items into one, for example, combining coded data and metadata into one data, and imply a method of “associating” as described above.

[0246] Furthermore, the implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the spirit of this technology.

[0247] For example, a configuration described as a single device (or processing unit) can be divided and configured into multiple devices (or processing units). Conversely, a configuration described above as multiple devices (or processing units) can be collectively configured into a single device (processing unit). Furthermore, configurations other than those described above can be added to the configuration of each device (or each processing unit). Additionally, provided that the overall system configuration and operation are substantially the same, a portion of the configuration of a particular device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0248] Furthermore, for example, the above program can be executed in any device. In this case, only that the device has the necessary functions (function blocks, etc.) and can obtain the necessary information is required.

[0249] Furthermore, for example, each step in a flowchart can be executed by a single device, or it can be shared and executed by multiple devices. Additionally, in cases where a step includes multiple processes, the multiple processing elements can be executed by a single device, or they can be shared and executed by multiple devices. In other words, multiple processing elements included in a single step can also be executed as processing elements of multiple steps. Conversely, processes described as multiple steps can be executed together as a single step.

[0250] Furthermore, for example, in a program executed by a computer, the processing of the steps describing the program can be performed sequentially in the order described in this specification, or it can be performed individually or in parallel at necessary timing points, such as when a call is made. That is, as long as there is no contradiction, the processing of each step can be performed in an order different from the above-described order. In addition, the processing of the steps describing the program can be performed in parallel with the processing of another program, or it can be combined with the processing of another program.

[0251] Furthermore, for example, multiple technologies related to this technology can be independently implemented as a single entity, as long as there is no contradiction. Of course, multiple arbitrary technologies can be combined to implement each other. For example, some or all of the technologies described in any embodiment can be implemented in combination with some or all of the technologies described in other embodiments. Furthermore, some or all of the aforementioned technologies can be implemented in combination with other technologies not described above.

[0252] Note that this technology can also be configured as follows.

[0253] (1) An information processing device, comprising:

[0254] A file generation unit, configured to generate a file, wherein the file is stored as follows:

[0255] A bitstream track, which stores a bitstream containing sub-pictures within a frame, and an extraction base track, which stores sub-picture specification information, which specifies the sub-picture to be extracted from the bitstream track.

[0256] (2) The information processing apparatus according to (1), wherein,

[0257] The specified information for the sub-image includes tile region identification information, which is the identification information of the tile region including the sub-image.

[0258] (3) The information processing apparatus according to (2), wherein,

[0259] The extraction of the basic track sets the sub-images extracted as a bitstream into a set, and stores the specified information of the sub-images for each set.

[0260] (4) The information processing apparatus according to (3), wherein,

[0261] The extracted basic track also stores set identifier information as identifier information of the set.

[0262] (5) The information processing apparatus according to any one of (2) to (4), wherein,

[0263] The bitstream track also stores, for each of the sub-images stored in the bitstream track, sub-image identification information as identification information of the sub-image and tile region identification information including the tile region of the sub-image.

[0264] (6) The information processing apparatus according to (5), wherein,

[0265] The bitstream track stores only the sub-image identification information for sub-images whose sub-image identification information matches the tile region identification information.

[0266] (7) The information processing apparatus according to any one of (1) to (6), wherein,

[0267] The extracted basic track also stores parameter set update information, which is used to update the parameter set to correspond to the bitstream of the extracted sub-image.

[0268] (8) The information processing apparatus according to (7), wherein,

[0269] The parameter set update information includes sub-image update information, which is used to update the sub-image identifier information, which serves as the identifier information of the sub-image, to correspond with the extracted bitstream of the sub-image.

[0270] (9) The information processing apparatus according to any one of (1) to (8), wherein:

[0271] The file also includes a merged base track, which stores information relating to the merging of multiple sub-images extracted from different bitstream tracks based on the extracted base track.

[0272] (10) The information processing apparatus according to any one of (1) to (9), wherein,

[0273] The content refers to point clouds, which represent objects with three-dimensional shapes as sets of points.

[0274] The sub-image is configured from a portion of the point cloud that can be independently decoded.

[0275] (11) An information processing method, comprising:

[0276] Generate a file, which is then stored as follows:

[0277] A bitstream track, which stores a bitstream containing sub-pictures within a frame, and an extraction base track, which stores sub-picture specification information, which specifies the sub-picture to be extracted from the bitstream track.

[0278] (31) An information processing apparatus, comprising:

[0279] A bitstream generation unit is configured to extract sub-images from a bitstream track based on a file-based extraction base track and generate a bitstream, wherein the file is stored as follows:

[0280] The bitstream track, which stores the bitstream containing the content of the sub-image within the frame, and

[0281] The extraction of the basic track stores sub-image specification information, which is used to specify the sub-image to be extracted from the bitstream track.

[0282] (32) The information processing apparatus according to (31), wherein,

[0283] The specified information for the sub-image includes tile region identification information, which is the identification information of the tile region including the sub-image.

[0284] (33) The information processing apparatus according to (32), wherein,

[0285] The extraction of the basic track sets the sub-images extracted as a bitstream into a set, and stores the specified information of the sub-images for each set.

[0286] (34) The information processing apparatus according to (33), wherein,

[0287] The extracted basic track also stores set identifier information as identifier information of the set.

[0288] (35) The information processing apparatus according to any one of (32) to (34), wherein,

[0289] The bitstream track also stores, for each of the sub-images stored in the bitstream track, sub-image identification information as identification information of the sub-image and tile region identification information including the tile region of the sub-image.

[0290] (36) The information processing apparatus according to (35), wherein,

[0291] The bitstream track stores only the sub-image identification information for sub-images whose sub-image identification information matches the tile region identification information.

[0292] (37) The information processing apparatus according to any one of (31) to (36), wherein,

[0293] The extracted basic track also stores parameter set update information, which is used to update the parameter set to correspond to the bitstream of the extracted sub-image.

[0294] (38) The information processing apparatus according to (37), wherein,

[0295] The parameter set update information includes sub-image update information, which is used to update the sub-image identifier information, which serves as the identifier information of the sub-image, to correspond with the extracted bitstream of the sub-image.

[0296] (39) The information processing apparatus according to any one of (31) to (38), wherein:

[0297] The file also includes a merged base track, which stores information relating to the merging of multiple sub-images extracted from different bitstream tracks based on the extracted base track.

[0298] (40) The information processing apparatus according to any one of (31) to (39), wherein,

[0299] The content refers to point clouds, which represent objects with three-dimensional shapes as sets of points.

[0300] The sub-image is configured from a portion of the point cloud that can be independently decoded.

[0301] (41) An information processing method, comprising:

[0302] Based on the extraction of the basic track from the file, sub-images are extracted from the bitstream track and a bitstream is generated. The file is stored as follows:

[0303] The bitstream track, which stores the bitstream containing the content of the sub-image within the frame, and

[0304] The extraction of the basic track stores sub-image specification information, which is used to specify the sub-image to be extracted from the bitstream track.

[0305] List of reference numerals

[0306] 300 Document generation device, 301 Control unit, 302 Document generation and processing unit, 311 Input unit, 312 Preprocessing unit, 313 Encoding unit, 314 Document generation unit, 315 Recording unit, 316 Output unit, 400 Client device, 401 Control unit, 402 Reproduction processing unit, 411 Document acquisition unit, 412 Document processing unit, 413 Decoding unit, 414 Display information generation unit, 415 Display unit, 416 Measurement unit, 417 Display control unit.

Claims

1. An information processing apparatus, comprising: The circuit is configured as follows: Generate a bitstream track, which is stored in a bitstream containing multiple sub-pictures within a frame, and Generate sub-image specification information, which is used to specify the sub-image to be extracted from the plurality of sub-images in the bitstream track. The sub-image specification information includes tile region identification information, which includes the identification information of the tile region of the sub-image to be extracted. as well as The generated file includes the bitstream track and the extracted basic track, wherein the extracted basic track includes the sub-image specified information. The circuit is further configured to store mapping information in the bitstream track by extending the sub-image identifier sample group information included in the bitstream track, the mapping information indicating the correspondence between the sub-image and the tile region.

2. The information processing apparatus according to claim 1, wherein, The circuit is configured to manage two or more sub-images among the plurality of sub-images extracted as a bitstream into a set, and to store specified information of the sub-images in the extraction base track based on each set.

3. The information processing apparatus according to claim 2, wherein, The extracted basic track also stores set identifier information as identifier information of the set.

4. The information processing apparatus according to claim 1, wherein, The bitstream track also stores, for each of the plurality of sub-images stored in the bitstream track, sub-image identification information as identification information of the sub-image and tile region identification information including the tile region of the sub-image.

5. The information processing apparatus according to claim 4, wherein, The bitstream track stores only the sub-image identification information for sub-images that correspond one-to-one with the tile region identification information.

6. The information processing apparatus according to claim 1, wherein, The extracted basic track also stores parameter set update information, which is used to update the parameter set to correspond to the bitstream of the extracted sub-image.

7. The information processing apparatus according to claim 6, wherein, The parameter set update information includes sub-image update information, which is used to update the sub-image identifier information, which serves as the identifier information of the sub-image, to correspond with the extracted bitstream of the sub-image.

8. The information processing apparatus according to claim 1, wherein: The file also includes a merge base track, which stores information relating to the merging of one or more sub-images extracted from two or more different bitstream tracks based on the extracted base track.

9. The information processing apparatus according to claim 4, wherein: The sub-image identifier sample group information indicates the sub-image identifier information included in the bitstream track in the decoding order. The circuit is also configured to manage the sub-image identification information in a sub-image sequence sample group stored in the bitstream track, the sub-image sequence sample group including rewrite information of the sub-image identification information.

10. An information processing method, comprising: Generate a bitstream track, which is stored in a bitstream containing multiple sub-pictures within a frame. Generate sub-image specification information, which is used to specify the sub-image to be extracted from the plurality of sub-images in the bitstream track. The sub-image specification information includes tile region identification information, which includes the identification information of the tile region of the sub-image to be extracted. as well as The generated file includes the bitstream track and the extracted basic track, wherein the extracted basic track includes the sub-image specified information. Furthermore, the sub-image identifier sample group information included in the bitstream track is extended to store mapping information in the bitstream track, and the mapping information indicates the correspondence between the sub-image and the tile region.

11. An information processing apparatus, comprising: The circuit is configured as follows: The control includes receiving files from bitstream tracks and extracting base tracks, wherein the bitstream tracks comprise bitstreams containing content with multiple sub-images. The region of the map to be reproduced is determined based on the bitstream track, and A bitstream is generated by extracting sub-images from the bitstream track based on the extraction base track. The extraction base track stores sub-image specification information, which specifies the sub-image to be extracted from among the plurality of sub-images in the bitstream track. The sub-image specification information includes tile region identification information, which identifies the tile region containing the sub-image to be extracted. The bitstream track includes mapping information by extending the sub-image identifier sample group information, and the mapping information indicates the correspondence between the sub-image and the tile region.

12. The information processing apparatus according to claim 11, wherein, The circuit is configured to manage two or more sub-images among the plurality of sub-images extracted as a bitstream into a set, and to store specified information of the sub-images in the extraction base track based on each set.

13. The information processing apparatus according to claim 12, wherein, The extracted basic track also stores set identifier information as identifier information of the set.

14. The information processing apparatus according to claim 11, wherein, The bitstream track also stores, for each of the plurality of sub-images stored in the bitstream track, sub-image identification information as identification information of the sub-image and tile region identification information including the tile region of the sub-image.

15. The information processing apparatus according to claim 14, wherein, The bitstream track stores only the sub-image identification information for sub-images that correspond one-to-one with the tile region identification information.

16. The information processing apparatus according to claim 11, wherein, The extracted basic track also stores parameter set update information, which is used to update the parameter set to correspond to the bitstream of the extracted sub-image.

17. The information processing apparatus according to claim 16, wherein, The parameter set update information includes sub-image update information, which is used to update the sub-image identifier information, which serves as the identifier information of the sub-image, to correspond with the extracted bitstream of the sub-image.

18. The information processing apparatus according to claim 17, wherein: The file also includes a merge base track, which stores information relating to the merging of one or more sub-images extracted from two or more different bitstream tracks based on the extracted base track.

19. The information processing apparatus according to claim 11, wherein: The sub-image identification information is stored in a sub-image sequence sample group stored in the bitstream track. The sub-image sequence sample group includes rewrite information of the sub-image identification information. The circuit is configured to, based on the mapping information, specify the sub-image identification information of the sub-image corresponding to the tile region to be reproduced, and The circuit is configured to rewrite the sub-image identification information in the sub-image sequential sample group.

20. An information processing method, comprising: The control includes receiving files from bitstream tracks and extracting base tracks, wherein the bitstream tracks comprise bitstreams containing content with multiple sub-images. The region of the map to be reproduced is determined based on the bitstream track, and A bitstream is generated by extracting sub-images from the bitstream track based on the extraction base track. The extraction base track stores sub-image specification information, which specifies the sub-image to be extracted from among the plurality of sub-images in the bitstream track. The sub-image specification information includes tile region identification information, which identifies the tile region containing the sub-image to be extracted. The bitstream track includes mapping information by extending the sub-image identifier sample group information, and the mapping information indicates the correspondence between the sub-image and the tile region.

Images