Information processing apparatus, information processing method, non-transitory computer-readable storage medium, and computer program product

Abstract

The invention relates to an information processing apparatus, an information processing method, a non-transitory computer-readable storage medium, and a computer program product. The information processing apparatus includes a storage control unit configured to cause a memory to store three-dimensional data and metadata corresponding to the three-dimensional data in a file in a predefined format, the storage control unit causes the memory to store a priority at the time of display in the file for each object included in the three-dimensional data or for at least a portion of the objects.

Description

Technical Field

[0001] This disclosure relates to information processing equipment, non-transitory computer-readable storage media, and computer program products, and particularly to information processing technology. Background Technology

[0002] Methods for generating 3D data using computer graphics are traditionally known. However, recently, methods have emerged for obtaining 3D data by scanning the shape and texture of objects (such as actual objects and people) using specialized equipment, studios, etc.

[0003] In recent years, there has been a trend of using 3D data obtained in this way for autonomous driving, driving support, or as data for free-viewpoint video to be displayed on display devices such as head-mounted displays.

[0004] On the other hand, since 3D data is usually very large, the Moving Picture Experts Group (MPEG) under the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) is working on standardizing standards for encoding 3D data and file format standards for storing encoded 3D data.

[0005] 3D data is typically processed in data formats such as point cloud data and 3D mesh data. However, in MPEG, for example, ISO / IEC 23090-5 Visual Volumetric Video Coding (V3C) and Video-Based Point Cloud Compression (V-PCC) (hereinafter referred to as V3C / V-PCC) (which are standards for encoding using video codecs, etc.) have been standardized as examples of standards for encoding point cloud data. Furthermore, ISO / IEC 23090-10 Visual Volumetric Video Coding Data Bearer (hereinafter referred to as V3C Bearer) has been standardized as a standard for storing point cloud data encoded by the aforementioned standards in files.

[0006] Incidentally, when using the aforementioned 3D data as data for free-viewpoint video, the rendering processing load during the display of 3D data can become problematic. For example, when watching team sports such as basketball or football in free-viewpoint video, the 3D data of multiple players scanned during the game must be rendered simultaneously. Furthermore, 3D data obtained by scanning people often requires high-quality data with high-resolution shape data and detailed textures, and because higher-quality data has a larger data volume, the processing load during rendering is often also greater.

[0007] Therefore, a method is needed to reduce the processing load used to display 3D data. Summary of the Invention

[0008] This disclosure provides a technique for advantageously performing processing for displaying 3D data and reducing processing load.

[0009] According to a first aspect of this disclosure, an information processing apparatus is provided, comprising: a storage control unit configured to cause a memory to store three-dimensional data and metadata corresponding to the three-dimensional data in a file in a predefined format, wherein the storage control unit causes the memory to store in the file a priority for each object included in the three-dimensional data or for at least a portion of the objects to be displayed.

[0010] According to a second aspect of this disclosure, an information processing apparatus is provided, comprising: an acquisition unit configured to acquire a display priority from a file storing three-dimensional data and metadata corresponding to the three-dimensional data in a predefined format, the display priority being stored for each object included in the three-dimensional data or for at least a portion of the objects; and a control unit configured to control the reproduction of the three-dimensional data based on the priority.

[0011] According to a third aspect of this disclosure, an information processing apparatus is provided, configured to generate metadata corresponding to three-dimensional data, the information processing apparatus comprising: an acquisition unit configured to acquire three-dimensional data, the three-dimensional data acquired by the acquisition unit including a plurality of objects; and a setting unit configured to set a display priority for each of the objects included in the three-dimensional data or for at least a portion of the objects, wherein the set priority is generated as metadata.

[0012] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer program instructions for using a computer as a unit of an information processing device.

[0013] According to a fifth aspect of this disclosure, a computer program product is provided that stores computer program instructions for causing a computer to function as a unit of an information processing device.

[0014] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is given by way of example. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments.

[0016] Figure 1 This is a block diagram illustrating an example of the hardware configuration of the information processing device 100.

[0017] Figure 2 This is a flowchart of the processing performed by the information processing device 100.

[0018] Figure 3 This is a diagram illustrating an example configuration for a 3D media file.

[0019] Figure 4 This is a diagram illustrating an example of metadata descriptions used to set the priority of an object.

[0020] Figure 5A It is a diagram used to explain the situation of viewing an object from a specific viewpoint.

[0021] Figure 5B It is a diagram used to explain the situation of viewing an object from a specific viewpoint.

[0022] Figure 5C It is a diagram used to explain the situation of viewing an object from a specific viewpoint.

[0023] Figure 6A This is a diagram illustrating an example of metadata descriptions used to set the priority of objects in association with a viewport.

[0024] Figure 6B This is a diagram illustrating an example of metadata descriptions used to set the priority of objects in association with a viewport.

[0025] Figure 7 This is a flowchart of the processing performed by the information processing device 100.

[0026] Figure 8 This is a diagram illustrating a scenario where a portion of an object can be divided into multiple sub-objects.

[0027] Figure 9 This is a diagram illustrating the view of an object that has been subdivided into tiles (the segmented object). Detailed Implementation

[0028] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are required, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0029] (First Embodiment)

[0030] First, refer to Figure 1The following block diagram illustrates an example of the hardware configuration of the information processing device 100 according to this embodiment. Computer devices such as personal computers (PCs), tablet terminals, or smartphones can be applied to the information processing device 100 according to this embodiment.

[0031] The CPU 102 uses computer programs and data stored in the RAM 103 to perform various processes. Therefore, the CPU 102 controls the operation of the entire information processing device 100 and executes or controls various processes described as being performed by the information processing device 100.

[0032] RAM 103 includes areas for storing computer programs and data loaded from ROM 104 and non-volatile memory 113, and areas for storing computer programs and data received from external devices via communication unit 107. Furthermore, RAM 103 includes working areas used by CPU 102, acquisition unit 108, analysis unit 109, setting unit 110, processing unit 111, and encoding unit 112 when performing various processes. Therefore, RAM 103 can appropriately provide various areas.

[0033] ROM 104 stores the configuration data of the information processing device 100, computer programs and data related to the startup of the information processing device 100, and computer programs and data related to the basic operation of the information processing device 100.

[0034] The operation input unit 105 is a user interface such as a keyboard, mouse and touch panel, and various instructions and information can be input to the information processing device 100 by user operation.

[0035] The display unit 106 includes a liquid crystal display or a touch panel display, and can display the processing results of the CPU 102 by using images, characters, etc. The display unit 106 can be a projection device such as a projector for projecting images and characters.

[0036] Furthermore, the configuration allows the display unit 106 to be a touch panel and the operation input unit 105 to include a touch panel sensor. In this case, when an operation input to the user interface screen displayed on the display unit 106 is detected, the operation input unit 105 outputs a control signal indicating the operation input to the CPU 102.

[0037] Communication unit 107 communicates with external devices via a network such as a LAN or the Internet. For example, communication unit 107 is a wired LAN Ethernet® PHY and MAC (transmission medium control processing). Furthermore, when information processing device 100 can be connected to a wireless LAN, communication unit 107 includes a controller, RF circuitry, and an antenna for performing wireless LAN control such as IEEE 802.11a / b / g / n / ac.

[0038] The corresponding operations of the acquisition unit 108, analysis unit 109, setting unit 110, processing unit 111, and encoding unit 112 will be described later. In this embodiment, the case where the acquisition unit 108, analysis unit 109, setting unit 110, processing unit 111, and encoding unit 112 are all implemented in hardware will be described. The acquisition unit 108, analysis unit 109, setting unit 110, processing unit 111, and encoding unit 112 can each be a separate piece of hardware, or two or more functional units can be implemented by a single piece of hardware. Furthermore, one or more of the acquisition unit 108, analysis unit 109, setting unit 110, processing unit 111, and encoding unit 112 can be implemented in software (computer program). In this case, the CPU 102 executes the computer program corresponding to the functional unit to implement the function corresponding to that functional unit.

[0039] Non-volatile memory 113 is, for example, flash memory such as an SD card or SSD, or magnetic recording device such as a hard disk drive. Non-volatile memory 113 stores operating system (OS), computer programs and data for causing CPU 102 to execute or control the various processes described as to be performed by information processing device 100.

[0040] CPU 102, RAM 103, ROM 104, operation input unit 105, display unit 106, communication unit 107, acquisition unit 108, analysis unit 109, setting unit 110, processing unit 111, encoding unit 112, and non-volatile memory 113 are all connected to the system bus 101.

[0041] Next, according to Figure 2 The flowchart describes a series of processes used to generate a 3D media file by detecting objects included in three-dimensional data (3D data) and storing metadata associated with the detected objects and encoded data obtained by encoding the 3D data in a predetermined format in a file.

[0042] In step S201, the obtaining unit 108 obtains 3D data including a point cloud or 3D mesh of the defined object. The method by which the obtaining unit 108 obtains the 3D data is not limited to a specific method. For example, the obtaining unit 108 may obtain 3D data received from an external device by the communication unit 107, or it may obtain 3D data pre-stored in the non-volatile memory 113. Since the 3D data is dynamic data whose content can change along the time axis and is generated for each frame, in step S201, the obtaining unit 108 obtains 3D data for multiple frames.

[0043] In step S202, the obtaining unit 108 loads (stores) the 3D data of the first frame from the 3D data of each frame obtained in step S201 into RAM 103. The method or configuration for obtaining the 3D data of each frame into RAM 103 is not limited to any specific method or configuration, as long as the 3D data of each frame in multiple frames can be processed by steps S203 to S209.

[0044] In step S203, the analysis unit 109 analyzes the 3D data of the frame (hereinafter referred to as the target frame) stored in the RAM 103 and determines whether the 3D data includes an object. The object is not limited to any particular type of object. The analysis unit 109 may, for example, use AI techniques such as machine learning to determine whether the 3D data includes an object.

[0045] As a result of this determination, if an object is included in the 3D data of the target frame (if an object is detected in the 3D data of the target frame), the process proceeds to step S205 via step S204.

[0046] On the other hand, if the object is not included in the 3D data of the target frame (if no object is detected in the 3D data of the target frame), the process proceeds to step S208 via step S204.

[0047] In step S205, the processing unit 111 assigns a unique identifier to each object included in the 3D data of the target frame. Each object included in the 3D data is assigned a different identifier than the others, and the same identifier is assigned to the same object between frames.

[0048] In step S206, the analysis unit 109 sets bounding boxes around each object included in the 3D data of the target frame. "Setting bounding boxes" means setting the position (3D coordinates) and size (vertical, horizontal, and height) of the bounding boxes in three-dimensional space. For example, the analysis unit 109 sets the center coordinates of the object as the position of the bounding box, sets the area extending from that position in the three axes X, Y, and Z as the bounding box surrounding the object, and sets the length of the bounding box in the corresponding X, Y, and Z axis directions as the size of the bounding box. The analysis unit 109 can also define the rotation angle of the bounding box relative to the three axes X, Y, and Z. Therefore, the bounding box may not necessarily be composed of planes perpendicular to the three axes X, Y, and Z. Furthermore, the bounding box does not need to be a region encompassing the entire object; it can be a region enclosing a portion of the object.

[0049] In step S207, the setting unit 110 sets the display priority for each object included in the 3D data of the target frame or for the bounding box of the corresponding object. The method for setting the priority for an object or for the bounding box is not limited to a specific method, and for example, the priority input by the user through operation input unit 105 can be set.

[0050] In this embodiment, as an example, priority indicates a priority level or recommendation level. Typically, priority objects are set to a higher priority level. Furthermore, the content (object) creator's intent is prioritized, and a higher recommendation level is set for content with a higher importance level. Priority is not limited to being set in step S207, but can be set before step S207.

[0051] Then, processing unit 111 performs standard data processing on the identifier assigned to the object, the position and size of the bounding box, and the priority set for the object or bounding box, conforming to a specified file format, and generates "metadata" that includes this information and is "in a specified form conforming to the specified file format." This metadata is metadata related to the 3D data obtained in step S201.

[0052] In step S208, the analysis unit 109 determines whether the 3D data of all frames has been analyzed. As a result of this determination, if the 3D data of all frames has been analyzed, the process proceeds to step S210, and if there is 3D data of frames that have not yet been analyzed, the process proceeds to step S209.

[0053] In step S209, the obtaining unit 108 loads (stores) the 3D data of the frame after the target frame in the 3D data of each frame obtained in step S201 into RAM 103.

[0054] In step S210, the encoding unit 112 encodes the 3D data of all frames using a method that conforms to an encoding standard (such as V3C / V-PCC), thereby generating encoded data of the 3D data.

[0055] In step S211, processing unit 111 stores the encoded data generated in step S210 and the metadata generated in step S207 in a file conforming to a file format standard such as V3C bearer (for storage control). Then, processing unit 111 outputs such a file as a 3D media file. The output destination of the 3D media file is not limited to a specific output destination; for example, processing unit 111 can output the 3D media file to non-volatile memory 113, or it can be sent to an external device via communication unit 107.

[0056] Next, refer to Figure 3 An example describing the configuration of a 3D media file generated by the information processing device 100. Figure 3 The file format described herein will be based on the ISO Basic Media File Format (hereinafter referred to as ISOBMFF), which is the basic specification for media files standardized by MPEG.

[0057] exist Figure 3 In the context of moov 301, there are atlas track 302, geometry track 310, attribute track 311, occupancy track 312, and metadata track 313.

[0058] Atlas track 302 stores, for example, metadata related to the region containing encoded data (encoded 3D data) stored in 3D media files, configuration information of the encoded data, etc.

[0059] The geometry track 310 is used to manage coordinate information indicating the shape of an object, and the attribute track 311 is used to manage attribute information including information such as the object's surface color and light reflectivity. The occupancy track 312 is used to manage information identifying the three-dimensional space in which an object exists, and the metadata track 313 is a track that can store attribute information such as 3D data that changes along the time axis.

[0060] Furthermore, these four tracks (geometric track 310, attribute track 311, occupancy track 312, and metadata track 313) are associated with atlas track 302 via track reference 305. The v3vg 306, v3va 307, v3vo 308, and cdsc 309 included in track reference 305 indicate the reference type of the corresponding associated track.

[0061] Incidentally, Figure 3An example of track configuration is shown when the point cloud of a defined object is encoded using the V3C / V-PCC encoding standard, and the data managed by the corresponding track (such as encoded data) is stored in mdat 315.

[0062] In this embodiment, metadata track 313 is used to store the priority of an object or bounding box as metadata associated with the object or bounding box in the 3D media file. Figure 4 This is a diagram illustrating an example of metadata descriptions used to set the priority of an object.

[0063] exist Figure 4 In the V3C payload, DynamicVolumetricMetadataSampleEntry 401 is one of the sample entries defined in the payload, and this sample entry can be used as... Figure 3 Metadata sample entry 314 is included in metadata track 313. In this case, the spatial region containing 3D data transmitted by atlas track 302 is considered a dynamic region.

[0064] Based on V3C's capabilities, when scene object information indicating the state of objects in 3D space changes over time, V3CVolumetricMetadataSample 402 can indicate changes in the position and size of the bounding box surrounding the detected objects. This sample serves as... Figure 3 Metadata sample 316 is stored in mdat 315, and there is one or more V3CVolumetricMetadataSample 402 when scene object information changes.

[0065] Here, in this embodiment, for example, a parameter called priority_value 403 is added to V3CVolumetricMetadataSample 402. In V3CVolumetricMetadataSample 402, num_region indicates the number of bounding boxes surrounding the detected object, and the region includes the location and size of these bounding boxes, the identifier of the object included in the bounding box, etc. Therefore, the value of priority_value 403 is a numerical value representing the priority of the corresponding bounding box. The value of priority_value 403 can be defined such that, for example, 0 (zero) is the highest priority, and the priority decreases as the value increases.

[0066] So far, the case where DynamicVolumetricMetadataSampleEntry 401 is stored as metadata sample entry 314 in metadata track 313 has been described. On the other hand, the case where V3CSpatialRegionCollectionBox 404 included in DynamicVolumetricMetadataSampleEntry 401 is directly stored as V3C atlas sample entry 304 in sample entry 303 of atlas track 302 will be described.

[0067] According to the V3C bearer, in this case, since the spatial region containing 3D data sent by the atlas track 302 is considered a static region, the scene object information representing the state of objects in 3D space does not change over time. Therefore, in this embodiment, the priority for each bounding box is defined, for example, by adding the parameter priority_value 405 to the V3CSpatialRegionCollectionBox 404.

[0068] Therefore, by defining priorities for each bounding box surrounding the detected object, during the processing for reproducing the object, these objects can be displayed starting from the highest priority, based on, for example, the processing capability of the device performing the reproduction processing.

[0069] Furthermore, when there are objects that need to be excluded from the reproduction process, the object can be excluded from the target of the reproduction process, for example, by setting priority_value403 or priority_value405 to a value corresponding to the lowest priority.

[0070] Reference Figure 3 The configuration file description already describes a multi-track configuration in moov 301 that includes multiple tracks (atlas track 302, geometry track 310, attribute track 311, occupancy track 312, and metadata track 313). However, depending on the V3C host, a single-track configuration where multiple tracks managing encoded 3D data are combined into one track, in addition to the metadata track 313, is also possible.

[0071] In a single-track configuration, only one track, referred to as the V3C bitstream track, is generated (where atlas track 302, geometry track 310, attribute track 311, and occupancy track 312 are combined into one track), and multiple different samples are defined in the sample entries within the V3C bitstream track. Different types of samples are managed by disposing of combined samples (where samples of different types with the same rendering time are combined into one sample within the V3C bitstream track).

[0072] In this single-track configuration, the multi-track configuration described above can be replaced by using a cdsc-type track reference to associate the V3C bitstream track with the metadata track.

[0073] Next, refer to Figures 5A to 5C Describes the situation of viewing an object from a specific viewpoint. Figure 5A This illustration shows the state of viewing a group of objects (objects 504, 505, and 506) defined by 3D data from viewpoint 501 in the gaze direction 502 and the viewpoint 503. In the following text, the combination of viewpoint 501, gaze direction 502, and viewpoint 503 will be referred to as viewport A.

[0074] The three objects mentioned above, which are the viewing targets in viewport A, are objects detected in the 3D data through the analysis and processing in step S203. Furthermore, Figure 5B Bounding boxes 507 to 509 are bounding boxes set for objects 504 to 506 respectively in step S206.

[0075] Here, when objects 504 and 505 are important objects as the viewing target, while object 506 is not important as the viewing target, for example, Figure 5C As illustrated in viewpoint 510, objects 504 and 505, which are important viewing targets, may be blocked by an unimportant object 506. In other words, when viewing viewport A, it is desirable to be able to view objects 504 and 505 that are not obscured by object 506, just as in viewpoint 511, by eliminating object 506 that interferes with viewing.

[0076] On the other hand, Figure 5A In viewport B, when viewing the aforementioned group of objects which has a different viewpoint, gaze direction, and angle of view than viewport A, object 506 does not interfere with viewing and therefore does not need to be eliminated. In other words, when viewing the same group of objects in different viewports, even at the same timing, the recommended and unrecommended objects can change depending on the viewport.

[0077] Therefore, reference Figure 6A and Figure 6B Describes a method that associates a viewport with the priority of an object. Figure 6A and Figure 6B This is a diagram illustrating an example of metadata descriptions used to set the priority of objects in relation to a viewport. Figure 6A and Figure 6B An example of metadata description for setting the priority of objects in association with viewport information is shown, where the viewport information includes information related to the viewpoint, gaze direction, and perspective in space based on 3D data.

[0078] ViewportInfoSampleEntry 601 is one of the sample entries defined in the V3C bearer, and this sample entry can be used as... Figure 3 Metadata sample entry 314 is included in metadata track 313. (See reference) Figure 4 The configuration described can coexist with the configuration described below, but in this case, metadata track 313 needs to be a different track.

[0079] In this embodiment, description A 603 is added to the ViewportInfoConfigurationBox 602 included in ViewportInfoSampleEntry 601. Description A 603 defines the recommended / non-recommended for each object.

[0080] num_of_objects indicates the number of recommended / non-recommended objects among the objects included in the viewport.

[0081] `object_deprecation_flag` is a flag indicating whether an object is a recommended or non-recommended object. For example, when `object_deprecation_flag` is 0, it means that subsequent objects with the identifier are recommended objects. On the other hand, when `object_deprecation_flag` is 1, it means that subsequent objects with the identifier are non-recommended objects.

[0082] The intent of `soi_object_idx[i]` is the identifier of each object included in the Scene Object Information Supplemental Enhancement Information (SEI) defined in V3C / V-PCC. That is, by using `object_deprecation_flag`, it is not necessary to describe the identifiers of all objects, especially when there are many objects, thus reducing the amount of description. The description method described here is merely an example and only needs to include the information used to identify the recommended / non-recommended objects.

[0083] When you add description B 604 instead of description A 603 to ViewportInfoConfigurationBox 602, you can define the priority level of each object. The priority_value of description B 604 is a parameter that indicates the priority level.

[0084] In addition, when defining recommended / non-recommended values ​​for the bounding boxes surrounding an object, this can be achieved by adding a description such as description C605 to ViewportInfoConfigurationBox 602.

[0085] In addition, when defining priority levels for each bounding box surrounding an object, this can be achieved by adding a description such as description D 606 to ViewportInfoConfigurationBox 602.

[0086] Furthermore, based on the V3C bearer, when the viewport changes dynamically, viewport information can be defined as a sample of metadata track 313. ViewportInfoSample 607 is a sample of viewport information defined in the V3C bearer, and the viewport information is included in ViewportInfo 608.

[0087] In this embodiment, the recommended / non-recommended status of each object can be indicated by adding description E 609 to ViewportInfo 608. The parameters described in description E 609 function as described in description A. Furthermore, by replacing description E 609 with descriptions B 604, C 605, and D 606, the priority level of each object, the recommended / non-recommended status of each bounding box surrounding the object, and the priority level of each bounding box surrounding the object can be defined respectively.

[0088] As another way to define the priority of objects or the bounding boxes surrounding them, it is also possible to list the identifiers of objects or the bounding boxes surrounding them in order of priority (descending or ascending). Furthermore, objects or bounding boxes whose identifiers are not included in the list can be implicitly defined as having the lowest (or highest) priority. The same applies to recommendation levels. For example, when an unlisted object is not displayed because it has the lowest priority, there is no need to re-identify the objects included in the viewport. Since only the listed objects need to be displayed according to the specified priority, the processing load during display can be reduced, especially when there are many objects.

[0089] Furthermore, in free-viewpoint video, the viewport typically changes dynamically over time, but as referenced... Figure 6A and Figure 6B As described, viewport information can be described as a sample of metadata track 313. That is, by generating a timed metadata track that stores viewport information, the changes of the viewport over time can be defined. Therefore, if multiple viewports exist, a timed metadata track can be generated that stores viewport information corresponding to the number of viewports.

[0090] The timing metadata track that stores viewport information can be used to track specific objects. For example, in sports content where multiple athletes are competing simultaneously, a viewport focusing on a specific athlete can be defined. In this case, the specific athlete can be tracked by setting either soi_object_idx[i] or the region defined in V3CSpatialRegion.

[0091] Regarding ViewportInfoSample 607, since viewports corresponding to the value of num_viewports can be defined, viewports can be defined for each specific athlete. However, in applications such as motion content streaming, for example, if tracks are segmented for each specific athlete of interest, the viewing user can prioritize watching that specific athlete by only receiving the track that stores the viewport information of that particular athlete. Furthermore, a specific athlete can be prioritized for playback by setting a higher priority for that athlete and lower priority for other athletes and objects.

[0092] Furthermore, to facilitate control over the display / non-display of each object, the encoded 3D data of each object can be made into independent encoded 3D data. When storing multiple independent encoded data in a file, in reference... Figure 3 In the described file configuration, the track for managing encoded 3D data consists of five tracks, including atlas track 302 and associated geometry track 310, attribute track 311, occupancy track 312, and metadata track 313. However, these five tracks can be generated for each object to manage independently encoded data for each object.

[0093] When multiple objects composed of independently encoded 3D data are displayed in the same 3D space, matching is required using coordinate origins, axis directions, tilt, and scaling as references in 3D space. Typically, encoded 3D datasets reference coordinates in 3D space, and positional information from these coordinates can be used to represent the position, shape, and movement of objects. Therefore, as a simple method to match coordinate origins, capturing multiple objects in an environment with the same pre-set coordinate origin allows objects to have the same coordinate origin and scaling, even if they are independent as encoded 3D data. This is possible in 3D capture studios or using 3D capture systems installed in stadiums, primarily for capturing athletes.

[0094] On the other hand, when objects captured in different environments, computer-generated CG objects, etc., are displayed in the same 3D space, it is necessary to define global coordinates and share these global coordinates among multiple tracks that manage the encoded 3D data to be displayed in the same 3D space. That is, this can be achieved by generating or obtaining information such as the offset, tilt, and scaling of each object relative to the origin and axis directions in the shared global coordinates, and storing them in, for example, the object's metadata track.

[0095] In this way, when storing each object's independently encoded 3D data as a separate track, priority or recommendation levels can be defined at the track level rather than the object level. That is, priority or recommendation levels can be set for each track ID, which is the track's identification information. However, as mentioned above, when there are multiple tracks managing the same encoded 3D data, such as in... Figure 3 In examples such as these, the atlas track 302, which stores reference information for each track, can be set as a representative track, and the track ID and priority or recommendation level of the representative track can be set.

[0096] Incidentally, when content is viewed as free-viewpoint video, viewport information is intended to define the viewing point recommended by the content creator, and when viewing content on a head-mounted display, especially when no viewing coordinates are specified on the side of the head-mounted display, viewport information can be used as the default viewing coordinates. In this case, if multiple timing metadata tracks exist that store viewport information, it is unknown which track is recommended or has a high recommendation level.

[0097] Therefore, the simplest method is to implicitly define that a smaller track ID value corresponds to a higher or lower recommendation level for the timing metadata track storing viewport information. Alternatively, this can be achieved by creating a set of timing metadata tracks storing viewport information within a track-level metabox, for example as an EntityToGroupBox, and describing the information used to identify the recommendation or priority level.

[0098] Next, refer to Figure 8 The methods for configuring objects and bounding boxes are described in more detail. Figure 8 This is a diagram illustrating a scenario where a portion of an object can be divided into multiple sub-objects.

[0099] exist Figure 8 In the image, viewport C includes viewpoint 801, gaze direction 802, and viewing angle 803, and viewing angle 804 indicates an example of video that can be viewed from viewing angle 803. Furthermore, object 810 is composed of sub-objects 811 and 812, which can be decoded and rendered independently.

[0100] Here, only sub-object 811 is included. Figure 8 In the viewport C illustrated, since the data of the object needed for viewing is only sub-object 811, the bounding box 805 only defines the area that includes sub-object 811. Therefore, by making it possible to obtain only the data of sub-object 811, unnecessary data not used for viewing can be avoided.

[0101] Therefore, in order to obtain only the data needed for viewing, it is desirable to segment the object into multiple regions and encode each region. For example, V3C / V-PCC, one of the point cloud encoding standards, supports the function of segmenting the object into blocks and encoding the blocks to improve parallel processing and spatial random access, and allows for independent decoding of each segmented block. Here, "segmenting into blocks" refers to the state of dividing the object into multiple regions in the 3D space in which the object exists, in a plane horizontal or perpendicular to the coordinate axes, and the method of segmenting the object into blocks and encoding them in this way so that they can be decoded independently will be referred to as block encoding below.

[0102] exist Figure 8 In this process, by dividing the sub-objects 811 and 812 that constitute object 810 into corresponding independent blocks and encoding the blocks, the sub-objects 812 that are not included in the viewpoint 803 of viewport C do not require processing such as obtaining, decoding and rendering of encoded data.

[0103] Next, refer to Figure 9 The description describes the situation where the object is further subdivided and encoded into chunks. Figure 9 This is a diagram illustrating the view of an object that has been subdivided into blocks (the segmented object).

[0104] exist Figure 9 In this context, viewport D includes a viewpoint 901, a gaze direction 902, and a viewing angle 903, and is viewing object 910 from slightly above. Object 910 is divided into multiple regions in 3D space, and each region is encoded as a block. Here, the blocks required for viewing viewport D are those included (or even partially included) in the viewing angle, but blocks located in occluded areas (blind spots) that do not contain data to be rendered are not needed for viewing. That is, even if included in the viewing angle, the encoded data of occluded areas not used for rendering is not required for viewing.

[0105] here, Figure 9 This illustrates the state where object 910 is segmented into chunks obtained by dividing each edge into three segments, and each segmented chunk is encoded. Sub-object 911 is the set of chunks required for viewing from viewport D's perspective 903, while sub-object 912 is the set of chunks included in the occluded region of viewport D. Therefore, regarding object 910, all chunks are included in viewport D's perspective 903, but only sub-object 911 is required for rendering, and sub-object 912 is not needed.

[0106] In this way, by using an encoding mechanism (such as block encoding) that subdivides objects and can be partially decoded, data in areas not included in the viewpoint when viewing a particular viewport, as well as data in occluded areas, can be omitted, and only the data required for viewing can be processed efficiently.

[0107] In chunked encoding, identifiers typically used to identify individual chunks are assigned and included in the encoded data, and this is similar for V3C / V-PCC. Here, when viewing the desired viewport, in order to obtain only the... Figure 9 Strictly speaking, it is necessary to know the identifier of the block corresponding to sub-object 911. Therefore, as referenced... Figure 6A and Figure 6B The information can be obtained by associating a list of identifiers for one or more chunks required for viewing (rendering) with ViewportInfoSampleEntry 601 or ViewportInfoSample 607, which stores viewport information.

[0108] In addition, recommended / non-recommended or priority information can be specified for each sub-object, such as... Figure 6A Descriptions A603 or B604, etc., can be used to describe this. By explicitly specifying the sub-objects that are not needed for viewing, the playback side can easily identify which sub-objects need to be displayed during the display, and thus reduce the processing load.

[0109] Although embodiments have been described to date with reference to V3C / V-PCC as a 3D data encoding standard, the effect is not dependent on the specific encoding standard implemented in the system. That is, in addition to V3C / V-PCC, geometry-based point cloud compression (G-PCC) of ISO / IEC 23090-9 (also standardized by MPEG) or video-based dynamic mesh coding (V-DMC) of ISO / IEC 23090-29 (a 3D mesh encoding standard) can also be used. Other 3D encoding standards, such as Gaussian splashing, can also be used.

[0110] Similarly, in the embodiments described to date, the V3C bearer has been described as a file format standard, but the implementation system does not depend on a specific file format standard. For example, similar storage methods can be used to support other storage standards that support the encoding format of the encoded 3D data to be stored, such as the ISO / IEC 23090-18 geometry-based point cloud compressed data bearer, which is the standard used to store the aforementioned G-PCC encoded data.

[0111] Furthermore, public information that is independent of encoding standards (such as viewport information in 3D space) is defined in ISO / IEC 23090-7 immersive media metadata and referenced from file format standards (such as V3C carriers and geometry-based point cloud compressed data carriers). In other words, viewport information can also be interpreted as generalized information and is not dependent on a specific standard.

[0112] Next, according to Figure 7 The flowchart describes the process for reproducing 3D data extracted from a 3D media file generated by the methods described in the embodiments to date. Figure 7 This is a flowchart used to explain an example of the process for reproducing objects from 3D media files that store 3D data by using priority levels.

[0113] In the following text, the information processing device 100 will be described according to... Figure 7 The present invention is not limited to the processing of the flowchart, and devices other than the information processing device 100 can obtain 3D media files and perform the above-described reproduction processing.

[0114] In step S701, the obtaining unit 108 loads (obtains) the 3D media file stored in the non-volatile memory 113 into the RAM 103.

[0115] In step S702, the analysis unit 109 analyzes the configuration of tracks and other elements included in the 3D media file obtained in RAM 103.

[0116] In step S703, as a result of the analysis in step S702, the analysis unit 109 determines whether the 3D media file includes a track (viewport track) that stores viewport information.

[0117] As a result of this determination, if the 3D media file includes a viewport track, the process proceeds to step S704, and if the 3D media file does not include a viewport track, the process proceeds to step S707.

[0118] In step S704, the analysis unit 109 determines whether the 3D media file includes multiple viewport tracks. As a result of this determination, if the 3D media file includes multiple viewport tracks, the process proceeds to step S705; if the 3D media file does not include multiple viewport tracks, the process proceeds to step S706.

[0119] In step S705, the analysis unit 109 selects the viewport track with the highest recommendation level from among multiple viewport tracks. The recommendation level is determined by the methods defined in the various examples above, or, if no recommendation level is specifically defined, the track with the lowest track ID value among the viewport tracks can be selected.

[0120] In step S706, the analysis unit 109 sets the viewport track selected in step S705 or a single viewport track included in the 3D media file as the analysis target and analyzes the analysis target.

[0121] In step S707, the analysis unit 109 analyzes the priority of the object. (See reference...) Figure 6A and Figure 6B As described above, when priorities are defined for each bounding box surrounding an object rather than for each object, the analysis unit 109 analyzes the priority of the bounding boxes. Although objects will be described below, they are processed in a similar manner even if they are bounding boxes.

[0122] In step S708, CPU 102 obtains the priority of the object corresponding to the 3D data of the first frame.

[0123] In step S709, CPU 102 identifies the objects to be displayed / not displayed based on the priority obtained in step S708.

[0124] In step S710, CPU 102 extracts encoded data from the 3D media file containing the object (display target object) identified in step S709 as the object to be displayed. This process is used to extract the data required to display the object from the 3D media file for the corresponding frame.

[0125] In step S711, CPU 102 decodes the encoded data extracted from the 3D media file in step S710 to obtain 3D data, and generates (renders) an image of the target object based on the 3D data.

[0126] In step S712, the CPU 102 determines whether processing has been completed for all frames included in the 3D media file. As a result of this determination, if processing has been completed for all frames, then according to... Figure 7 The processing of the flowchart ends. On the other hand, if there are frames whose processing has not yet been completed, the processing proceeds to step S713.

[0127] In step S713, CPU 102 determines whether the object's priority has changed dynamically based on the analysis result in step S702. If the object's priority has changed dynamically, the process proceeds to step S714; otherwise, the process proceeds to step S709. In step S714, CPU 102 obtains the object's priority in the next frame.

[0128] The descriptions to date have typically described how information related to object priority or recommendation levels, as well as viewport-related information, is stored in files. However, viewing-related information does not necessarily need to be stored in files. That is, refer to... Figure 4 , Figure 6A and Figure 6B Information related to priority or recommendation levels, as well as viewport-related information, can be stored in a different location than the file storing the encoded data.

[0129] For example, metadata information associated with encoded data can be created as a file and associated with it in a software-managed system, or the information can be stored in a storage device such as RAM without being created as a file.

[0130] This format is particularly suitable for large-scale systems, and efficient centralized management becomes possible by storing priority information associated with viewports together rather than in a single file.

[0131] Incidentally, ISO / IEC 12113:2022 glTF 2.0 is known as a format for forwarding 3D content and describing 3D scene information. Furthermore, in MPEG, ISO / IEC 23090-14 scene description is standardized by extending glTF 2.0. ISO / IEC 23090-14 scene description is a technique that allows data encoded by MPEG-standardized 3D or audio coding standards to be defined in 3D scene information. Therefore, object position information and viewport information can be defined as 3D scene information by applying glTF 2.0 or scene description standards. By using such a standard for definition, this information can be expected to be used in systems that support glTF 2.0.

[0132] In the descriptions so far, the terms priority and recommendation have been used as semantically similar terms. However, as mentioned above, they are used to indicate scales set for different intentions, where the former is intended for content that is generally prioritized, while the latter focuses on the intention of the content creator. In this embodiment, since the intention that defines the scale is not important, the two terms (priority and recommendation) can be interpreted as synonyms.

[0133] Furthermore, the descriptions to date have outlined forms for defining priorities for individual objects or for the bounding boxes surrounding objects. However, for example, when multiple objects are in contact or entangled in a complex manner, it may be difficult to clearly distinguish and define the bounding boxes surrounding the respective objects. Therefore, for example, there may be cases where multiple objects are included within a single bounding box. That is, it is conceivable that the objects to be reproduced cannot be separated at the 3D data level. Therefore, in such cases, it is desirable to assign identifiers to "bounding boxes surrounding one or more objects" and set priorities.

[0134] Furthermore, by using priorities for each object and for each bounding box surrounding the object, when reproducing content that incurs a high processing load and uses 3D data on a display terminal with low processing power, for example, the content can be displayed sequentially starting from the highest priority within the processing power of the display terminal. Therefore, the load of rendering 3D data can be reduced and fault-free reproduction processing can be performed.

[0135] Furthermore, it is possible to display objects with low priority or those that might interfere with viewing when viewed from a specific field of view, while in free-viewpoint video. Low-priority objects are not simply hidden; for example, only the wireframe of the 3D object can be displayed, and in the case of scalable 3D data such as Level of Detail (LoD), the level of detail displayed can be controlled according to priority, thereby enabling advantageous rendering processing and reducing processing load. Furthermore, it is conceivable to use methods that control the priority of objects to be rendered based on the user's permissions when viewing the 3D data.

[0136] Furthermore, various definitions can be applied to the recommendation levels and recommended / not recommended in this embodiment. For example, the recommendation level can be a parameter assuming an integer value from 0 (lowest recommendation level) to 100 (highest recommendation level). Additionally, a recommendation level greater than or equal to a threshold can represent "recommended," and a recommendation level less than the threshold can represent "not recommended." Furthermore, the recommendation level can be represented using binary values ​​"0" and "1," where a recommendation level "0" represents "not recommended," and a recommendation level "1" represents "recommended."

[0137] Rendering high-quality 3D data involving multiple people on devices (such as head-mounted displays, smartphones, or tablet PCs) for viewing as free-viewpoint video places stringent demands on hardware resources (such as CPUs and memory). Therefore, services known as cloud rendering have emerged, where rendering is performed on computing devices in the cloud. However, in cloud rendering, data such as viewpoint, viewing direction, and angle of view are sent from each viewing terminal, and 2D images are sent as the rendering result to each viewing terminal; therefore, transmission latency can become a problem depending on the communication environment and use case. Furthermore, when there are many viewing terminals using cloud rendering, the increased load on the cloud-side computing devices can become a problem. By using the information processing device according to this embodiment, advantageous rendering can be achieved and the rendering load reduced.

[0138] (Second Embodiment)

[0139] In the first embodiment, the 3D media file generation process and the object reproduction process based on the 3D media file performed by the information processing device 100 have been described. However, these processes can be performed using multiple computer devices. Part of these processes can be executed by external devices (e.g., cloud servers), and subsequent processing can be performed by the information processing device 100 based on the results of the execution. Therefore, the configuration of the overall process processor and the system used is not limited to a specific form.

[0140] Furthermore, the computer program stored in the information processing device 100 can be downloaded from a homepage on the Internet to the non-volatile memory 113 by connecting to a homepage on the Internet from a browser on the information processing device 100, which acts as a client computer. The computer program can be an uncompressed computer program file or a compressed computer program file with automatic installation functionality.

[0141] Furthermore, the program code constituting the computer program can be divided into multiple files, and each file can be downloaded to the information processing device 100 from a different homepage. In other words, a WWW server from which multiple users download files of computer programs used to implement the aforementioned processing in the information processing device 100 can also be considered an example.

[0142] Furthermore, such a computer program can be distributed to users by encrypting it and storing it on a storage medium such as a CD-ROM, and users who have met predetermined conditions can download the decryption key information from a homepage via the Internet. In other words, users can install the encrypted computer program on the information processing device 100 by executing the encrypted program using the key information.

[0143] For the purpose of providing a specific explanation, the numerical values, processing timings, processing sequences, processing executors, data (information) configurations / acquisition methods / sending destinations / sending sources / storage locations, etc., used in the above embodiments have been given as examples, and the present invention is not intended to be limited to such examples.

[0144] Furthermore, some or all of the embodiments described above can be appropriately combined and used. Additionally, some or all of the embodiments described above can be selectively used.

[0145] Other embodiments

[0146] The embodiments of the present invention can also be implemented by the following method: providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0147] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing apparatus comprising: a storage control unit configured to cause a storage to store three-dimensional data and metadata corresponding to the three-dimensional data in a file in a predefined format, wherein the storage control unit causes the storage to store, in the file, a priority at a time of display for each object included in the three-dimensional data or for at least a part of the objects. 2.The information processing apparatus according to claim 1, wherein the storage control unit causes the storage to store, in the file, viewport information including a viewpoint, a gaze direction, and a viewing angle in a space based on the three-dimensional data. 3.The information processing apparatus according to claim 2, wherein the storage control unit causes the storage to store, in the file, each of a plurality of the viewport information as a different track. 4.The information processing apparatus according to claim 1, wherein the storage control unit sets each of the objects included in the three-dimensional data or at least a part of the objects as a target, and causes the storage to store, in the file, information that can be used by a target to be identified from the target to be recommended for display. 5.The information processing apparatus according to claim 1, wherein the storage control unit sets each of the objects included in the three-dimensional data or at least a part of the objects as a target, and causes the storage to store, in the file, information that can be used by a target to be identified from the target to be not recommended for display. 6.The information processing apparatus according to claim 1, wherein the priority is a level of priority or a recommended level set according to a level of importance of content. 7.The information processing apparatus according to claim 4, wherein the storage control unit causes the storage to store, in the file, each of the objects included in the three-dimensional data as a different track. 8.The information processing apparatus according to claim 1, further comprising: an encoding unit configured to encode the objects by using a predetermined encoding method, the encoding unit dividing the objects into a plurality of regions and encoding each of the regions as a divided object so as to be able to be decoded independently, and wherein the storage control unit sets a priority for at least a part of a plurality of the encoded objects. 9.The information processing apparatus according to claim 2, further comprising: an encoding unit configured to encode the objects by using a predetermined encoding method, wherein the encoding unit divides the objects into a plurality of regions and encodes each of the regions as a divided object so as to be able to be decoded independently, and the storage control unit sets a priority for recommended acquisition for an encoded object of the plurality of the encoded objects that is located in a region required for display from the viewport information, and sets a priority for not recommended acquisition or does not set a priority for an encoded object of the plurality of the encoded objects that is located in an occluded region of the viewport information. 10.An information processing apparatus comprising: an obtaining unit configured to obtain, from a file in which three-dimensional data and metadata corresponding to the three-dimensional data are stored in a predefined format, a display-time priority, the display-time priority being stored for each object included in the three-dimensional data or for at least a part of the objects; and a control unit configured to control reproduction of the three-dimensional data based on the priority.

11. The information processing apparatus according to claim 10, wherein a display target object can be obtained as a divided object that is one of the obtained plurality of regions that is divided, the priority is set for each divided object and is applied when the display target object is reproduced from a predetermined viewport, and in the control, a divided object that is located in a region required to display the display target object in the predetermined viewport is identified from the obtained priority of each divided object, and in the obtaining, the identified divided object is obtained.

12. The information processing apparatus according to claim 11, wherein the priority includes information that intends to recommend display and information that intends to not recommend display.

13. The information processing apparatus according to claim 10, wherein the obtaining unit obtains, from the file, viewport information that includes a viewpoint, a gaze direction, and a viewing angle in a space based on the three-dimensional data, and the control unit controls the reproduction based on the viewport information and the priority associated with the viewport information.

14. The information processing apparatus according to claim 13, wherein the obtaining unit obtains, from the file, viewport information having a highest recommendation level among a plurality of the viewport information.

15. The information processing apparatus according to claim 10, wherein information that can be utilized to identify a target that is recommended to be displayed from a corresponding object included in the three-dimensional data or from at least a part of the corresponding object is stored in the file.

16. The information processing apparatus according to claim 10, wherein information that can be utilized to identify a target that is not recommended to be displayed from a corresponding object included in the three-dimensional data or from at least a part of the corresponding object is stored in the file.

17. The information processing apparatus according to claim 10, wherein the priority is a level of priority or a recommendation level that is set according to a level of importance of content.

18. The information processing apparatus according to claim 10, wherein the obtaining unit obtains a track in which predetermined viewport information is stored, among a plurality of tracks, each track storing respective viewport information among a plurality of viewport information as different tracks, and the control unit selects a display target object according to the priority associated with the viewport information.

19. An information processing apparatus configured to generate metadata corresponding to three-dimensional data, the information processing apparatus comprising: an obtaining unit configured to obtain three-dimensional data, the three-dimensional data obtained by the obtaining unit including a plurality of objects; and a control unit configured to control reproduction of the three-dimensional data based on the priority. a setting unit configured to set, for each of the objects included in the three-dimensional data or for at least part of the objects, a priority at the time of display, wherein the set priority is generated as metadata.

20. An information processing method comprising: a storage control step of causing a storage to store three-dimensional data and metadata corresponding to the three-dimensional data in a file in a predefined format, wherein in the storage control step, the storage is caused to store, in the file, a priority at the time of display for each of the objects included in the three-dimensional data or for at least part of the objects.

21. An information processing method comprising: an obtaining step of obtaining a priority at the time of display from a file in which three-dimensional data and metadata corresponding to the three-dimensional data are stored in a predefined format, the priority at the time of display being stored for each of the objects included in the three-dimensional data or for at least part of the objects; and a control step of controlling reproduction of the three-dimensional data based on the priority.

22. An information processing method for generating metadata corresponding to three-dimensional data, the information processing method comprising: an obtaining step of obtaining three-dimensional data, the three-dimensional data obtained in the obtaining step including a plurality of objects; and a setting step of setting, for each of the objects included in the three-dimensional data or for at least part of the objects, a priority at the time of display, wherein the set priority is generated as metadata.

23. A non-transitory computer-readable storage medium storing computer program instructions for causing a computer to execute each step of the information processing method according to any one of claims 20 to 22.

24. A computer program product including computer program instructions for causing a computer to execute each step of the information processing method according to any one of claims 20 to 22. ​ ​

Images