Information processing device, reproduction device, information processing method, and program

Abstract

To operate a single file storing information for displaying a plurality of three-dimensional object data in the same coordinate system.SOLUTION: First data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object are acquired. Arrangement information for arranging the first three-dimensional object and the second three-dimensional object in the same coordinate system is generated. Meta data is common for the first three-dimensional object and the second three-dimensional object and meta data including information related to decryption of the first three-dimensional object and the second three-dimensional object is generated. A first track for managing first data, a second track for managing second data, and a third track for managing the meta data are generated. A single file storing the first track, the second track, the third track, the first data and the second data, the arrangement information and the meta data are generated.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a playback device, an information processing method, and a program. [Background technology]

[0002] A method using computer graphics has been known for generating 3D object data, but in recent years, a method has been developed in which 3D object data is obtained by scanning the shapes of real objects or people using dedicated equipment or studios.

[0003] In addition, data utilization is also being promoted by arranging multiple 3D object data generated or acquired in this way in the same three-dimensional space. For example, in the field of autonomous driving or driving assistance, progress is being made in the development of a mechanism for updating road information in real time by superimposing 3D object data around the road acquired by an in-vehicle remote sensing device called LiDAR (Light Detection And Ranging or Laser Imaging Detection and Ranging) on ​​a dynamic map.

[0004] Meanwhile, the Moving Picture Experts Group (MPEG), which is under the umbrella of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), is working on standardization of a standard for encoding 3D object data such as point clouds or meshes, and a file format standard for storing the encoded 3D object data.

[0005] Patent Document 1 discloses a technology that makes it possible to, for example, partially change the quality of 3D object data by spatially dividing 3D object data encoded using G-PCC (Geometry based Point Cloud Compression), which is being standardized by MPEG, and generating position information in three-dimensional space for each divided part of the 3D object data, as well as grouping information indicating that each divided part belongs to the same group. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 137642 Summary of the Invention [Problem to be solved by the invention]

[0007] However, although the technology described in Patent Document 1 discloses a form in which one piece of 3D object data is divided, it does not take into consideration a form in which a plurality of different pieces of 3D object data are stored in a single file.

[0008] Furthermore, glTF, a format for expressing 3D models, describes the structure and composition of a 3D space in JSON format, and it is possible to place multiple 3D object data in the same 3D space. However, in glTF, the association between the 3D space and the object data is performed by URI, and it was necessary to read and analyze each object data to determine whether the object data was in a displayable data format.

[0009] The present invention aims to operate a single file that stores information for displaying multiple pieces of three-dimensional object data in the same coordinate system. [Means for solving the problem]

[0010] In order to achieve the object of the present invention, for example, an information processing device according to one embodiment includes the following configuration: an acquisition means for acquiring first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object, a first generation means for generating arrangement information for arranging the first three-dimensional object and the second three-dimensional object in the same coordinate system, a second generation means for generating metadata common to the first three-dimensional object and the second three-dimensional object, the metadata including information related to decoding of the first three-dimensional object and the second three-dimensional object, a third generation means for generating a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata, and a fourth generation means for generating a single file storing the first track, the second track, and the third track, the first data, the second data, the arrangement information, and the metadata. Effect of the Invention

[0011] Operate a single file that stores information for displaying multiple 3D object data in the same coordinate system. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the first embodiment. [Diagram 2] 6 is a flowchart showing an example of a file generation process according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing the structure of a file generated by the information processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining arrangement information according to the first embodiment. [Diagram 5] FIG. 4 is a diagram for explaining light source information according to the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining visual field information according to the first embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of the functional configuration of the playback device according to the first embodiment. [Figure 8] 5 is a flowchart showing an example of a rendering process according to the first embodiment. [Figure 9] FIG. 13 is a diagram showing the structure of a file generated by a modified example of the information processing device. [Figure 10] FIG. 11 is a block diagram showing an example of a hardware configuration of an apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] [Embodiment 1] The functional configuration of an information processing device according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of an information processing device 100 according to embodiment 1. The information processing device 100 according to this embodiment includes a data acquisition unit 101, a conversion information generation unit 102, a data analysis unit 103, a data generation unit 104, a track generation unit 105, and a file storage unit 106.

[0015] The data acquisition unit 101 acquires encoded three-dimensional (3D) object data, such as encoded point cloud data or 3D mesh data. The data acquisition unit 101 can acquire various data from, for example, an external device (not shown). In this embodiment, the 3D object data acquired by the data acquisition unit 101 is encoded and has information (decoding information) used when decoding the 3D object data. The decoding information according to this embodiment is metadata including information related to decoding, and includes a parameter set (profile information) referred to by a decoder when decoding. In this embodiment, the "information related to decoding" includes information indicating whether the 3D object data is decodable as a parameter set, and whether the 3D object data is decodable is determined by analyzing this parameter set.

[0016] Note that the decoding information here is information to be referenced in decoding, such as a profile or level that identifies a toolset used in encoding, and can be added by a known encoding technique, so detailed description will be omitted. The data analysis unit 103 analyzes the acquired 3D object data and extracts the decoding information. Hereinafter, a 3D object represented by such 3D object data may be simply referred to as an "object."

[0017] The conversion information generating unit 102 generates placement information for placing the acquired 3D object data in an arbitrary three-dimensional space. The placement information according to this embodiment is 3D coordinate conversion information for converting the local coordinates of two or more 3D objects into coordinates in the same coordinate system, and a detailed description will be given later with reference to Figs. 3 and 4. In this embodiment, local coordinates of each 3D object are generated from each of the 3D object data acquired by the data acquiring unit 101, and these local coordinates are converted into global coordinates in the same coordinate system.

[0018] The data generation unit 104 generates metadata for managing the decryption information extracted by the data analysis unit 103. In this embodiment, the decryption information of the acquired multiple 3D object data is common, and the data generation unit 104 has a function of generating common metadata such as that shown in 303 in FIG. 3 described later. The common metadata according to this embodiment includes common information used for drawing each object and decryption information. This common information is, for example, light source information or field of view information, and details will be described later with reference to FIG. 5 and FIG. 6.

[0019] The track generating unit 105 according to the present embodiment generates a trak block (track), which will be described later, as a block for managing various data in a file generated by the information processing device 100. Here, the track generating unit 105 can generate a track for storing information on the acquired 3D object data, and a track for managing the common metadata generated by the data generating unit 104.

[0020] The file storage unit 106 generates a file that stores the acquired multiple pieces of 3D object data and the track generated by the track generation unit 105.

[0021] Next, a series of processes from acquisition of 3D object data to file generation by the information processing device 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of file generation processing performed by the information processing device according to this embodiment. The process in Fig. 2 is started when 3D object data is transmitted to the information processing device 100 from an external device (not shown), for example.

[0022] In S201, the data acquisition unit 101 acquires 3D object data. Here, the data acquisition unit 101 acquires the 3D object data from an external device. In S202, the track generation unit 105 generates a track for managing the acquired 3D object data. The track for managing the 3D object data will be described later with reference to FIG.

[0023] In S203, the data acquisition unit 101 determines whether or not all of the object data to be stored in the file in S201 has been acquired. If all of the object data has been acquired, the process proceeds to S204, and if not, the process returns to S201. Here, the data acquisition unit 101 may be set to acquire a predetermined number of pieces of 3D object data, for example, and may determine whether or not the predetermined number of pieces of 3D object data has been acquired. Also, for example, the data acquisition unit 101 may perform this determination by receiving a signal indicating whether or not all of the 3D object data has been transmitted from an external device that transmits the 3D object data.

[0024] In S204, the conversion information generating unit 102 sets an origin (reference position) in a coordinate system (global coordinate system) of an arbitrary three-dimensional space in which the acquired 3D object data is to be placed. This global coordinate system is a coordinate system used when displaying each 3D object in the coordinate system, and can be set arbitrarily. For example, the global coordinate system may be a coordinate system set based on geographical coordinates specified by a GPS (Global Positioning System). The global coordinate system may also be a coordinate system used to display the coordinates of each space defined by a space ID. In this case, a predetermined position in a space specified by a space ID in the global coordinate system (for example, one of the corners of a space such as a voxel specified by a space ID) is expressed as the reference point of the space ID.

[0025] The spatial ID according to this embodiment is identification information that specifies a spatial position in the same global coordinate system. That is, in this embodiment, since the spatial ID can uniquely specify any 3D spatial position in the real world, the spatial relationship specified by multiple spatial IDs can also be uniquely specified. Therefore, when using a spatial ID as a reference position, it is possible to set a reference point of a different spatial ID for each 3D object data as the reference position.

[0026] In addition, in this embodiment, it is assumed that all 3D objects to be processed are transformed into the same global coordinate system, but this is not particularly limited. For example, the 3D objects may be divided into several groups, and the local coordinates of each 3D object may be transformed into coordinates of a different three-dimensional coordinate system for each group.

[0027] In this embodiment, it has been described that the spatial ID specifies a spatial position in a common global coordinate system. However, each spatial ID may specify a position in a different three-dimensional coordinate system. For example, a spatial ID may be assigned to each floor of a structure, and a coordinate system with a corner of the floor as the origin may be referenced.

[0028] In S205, the conversion information generating unit 102 generates 3D coordinate conversion information. The 3D coordinate conversion information according to this embodiment includes information indicating the position in the global coordinate system, the orientation in the global coordinate system, and the size in the global coordinate system of the corresponding 3D object. Here, the 3D coordinate conversion information includes, for example, an offset of the local coordinate with respect to the origin of the global coordinate system (amount of deviation of the origin), an inclination (rotation angle) of the coordinate axis with respect to the global coordinate system, and scale information (enlargement / reduction ratio) with respect to the global coordinate system, which are calculated for each object.

[0029] In S206, the data analysis unit 103 analyzes the acquired 3D object data and extracts the decoding information. In S207, the data generation unit 104 generates common metadata including the decoding information used for decoding each 3D object data.

[0030] In S208, the track generating unit 105 generates a track (base track) that stores the 3D coordinate conversion information and the common parameters. In S209, the file storage unit 106 generates a file that stores the acquired 3D object data and the track generated by the track generating unit 105, and ends the process in FIG.

[0031] The format of the encoded 3D object data is not particularly limited as long as it contains geometry information, which is position information in a three-dimensional space. For example, the 3D object data used in this embodiment may be in the format of point cloud data or 3D mesh data.

[0032] Next, a specific example of a file generated by the series of processes described in Fig. 2 will be described with reference to Fig. 3. Fig. 3 is a schematic diagram showing an example of the internal structure of a file generated by the information processing device 100 according to this embodiment. Note that the file format described in Fig. 3 will be described as being in accordance with the ISO Base Media File Format (hereinafter, referred to as ISOBMFF), which is the basic specification of media files standardized by MPEG, but is not limited to this as long as data can be stored in the same manner.

[0033] The file 300 generated by the information processing device 100 according to this embodiment includes a plurality of boxes identified by four-character identifiers, and stores information in each box according to its purpose. Hereinafter, each box is represented by a four-character identifier assigned to the box. The file 300 according to this embodiment includes boxes ftyp, moov301, and mdat310. ftyp (FileTypeBox) has a four-character identifier called brand for identifying the type / subtype of the image file.

[0034] The moov (MovieBox) 301 includes multiple trak (TrackBox) boxes. The moov 301 according to this embodiment can store metadata related to data such as video or images, and the trak boxes (tracks) can store data indicating the arrangement positions (index information) of data such as video, images, or audio. Here, the moov 301 includes a total of nine tracks including the base track 302 and tracks 306 to 309. Here, the moov 301 stores geometry tracks 1 to 4 that manage coordinate information for indicating the shape of an object, and attribute tracks 1 to 4 that manage attribute information of the object, such as the surface color or light reflectance of the object.

[0035] In the example of Fig. 3, information corresponding to four independent objects 1 to 4 is stored in moov301, which are shown as geometry tracks 1 to 4 and attribute tracks 1 to 4, respectively. That is, for example, geometry track 1 (306) and attribute track 1 (307) are included as blocks that store information corresponding to object 1. Note that the association between the geometry track and the attribute track is defined by generating reference information to the attribute track in the geometry track.

[0036] The actual data of the coordinate information and attribute information managed by these tracks is stored in mdat (MediaDataBox) 310. That is, the information stored in moov 301 is index information indicating the reference destination of the actual data stored in mdat 310, and although mdat 310 is shown small in Fig. 3 for convenience of explanation, in terms of the size of the data stored, the corresponding data stored in mdat 310 is much larger than the information stored in each track.

[0037] The base track 302 stores common metadata 303 and 3D coordinate conversion information corresponding to each object. In the example of Fig. 3, the 3D coordinate conversion information 1 to 4 corresponding to the objects 1 to 4, respectively, is information for arranging the object data in global coordinates of a common three-dimensional space (hereinafter referred to as a virtual space). Here, local coordinates are managed as coordinate information in the geometry track for each object. Therefore, the 3D coordinate conversion information includes the offset of the local coordinates with respect to the origin of the global coordinates (amount of deviation of the origin), the inclination (rotation angle) of the coordinate axis with respect to the global coordinates, scale information (enlargement / reduction ratio) with respect to the global coordinate system, and the like, which are calculated for each object, as described above.

[0038] Here, a specific example of the 3D coordinate conversion information will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing an example of the 3D coordinate conversion information according to this embodiment. In Fig. 4, global coordinates 401 are a single spatial coordinate in the virtual space, and local coordinates 1 to 4 are unit vectors when expressing the coordinate systems of the local coordinates of objects 1 to 4, respectively. Local coordinate 1 (402) will be described below.

[0039] The origin offset 403 is information that indicates the position of the origin of the local coordinate 1 from the origin of the global coordinate 401 in the virtual space as an offset value, and can be indicated by offset values ​​(dx, dy, dz) for three axes in the three-dimensional Cartesian coordinate system shown in FIG.

[0040] 4 are information indicating the inclination (rotation angle) of the coordinate axis of the local coordinate system with respect to the global coordinate system 401. In the example of FIG. 4, this rotation angle is indicated by a roll angle / pitch angle / yaw angle in a three-dimensional orthogonal coordinate system, but a different method may be used as long as it can express the rotation angle, such as using a quaternion.

[0041] In the example of FIG. 4, when each 3D object data is represented in a virtual space, scale information (enlargement / reduction ratio) is added to set the size of the 3D object data to a desired ratio. The scale information may be set for each of the X, Y, and Z directions in each local coordinate system, for example. The value of the scale information may be defined as an integer or a fraction. The scale information may be set based on a user input, for example, or may be added to the 3D object data acquired by the data acquisition unit 101.

[0042] So far, we have explained about local coordinate 1, but it is possible to generate various 3D coordinate conversion information in the same way for the other local coordinates 2 to 4. By generating 3D coordinate conversion information for each object in this way, it is possible to determine relative position information for multiple 3D object data in the virtual space.

[0043] According to this configuration, since multiple different 3D object data can be stored in one file, it is possible to implement multiple 3D objects in combination using only one file. For example, background 3D object data and person 3D object data may be combined, or a part of a dynamic map expected to be used in autonomous driving may be combined with near real-time 3D object data acquired by LiDAR.

[0044] Each 3D object data may have a different frame rate, and furthermore, static 3D object data and dynamic 3D object data may be used in combination. When storing static 3D object data in a file, it is possible to store the data as an item in an iloc or iinf box (not shown) instead of in a trak box.

[0045] As shown in Fig. 3, the 3D coordinate transformation information is described as being stored in the base track, but the storage location is not particularly limited to this. For example, the 3D coordinate transformation information may be stored in tracks corresponding to each object, rather than in common metadata, such as being stored in each of geometrics 1 to 4.

[0046] Here, the virtual space in which the multiple 3D object data are arranged is assumed to be a virtual space set by the user. However, as described above, the global coordinates may be a coordinate system set based on geographical coordinates specified by a GPS, or may be a predefined coordinate system used when using a spatial ID.

[0047] Next, an example of metadata other than the decoding information stored in the common metadata 303 in FIG. 3 will be described with reference to FIG. 5 and FIG. 6. FIG. 5 is a schematic diagram showing an example of light source information of a virtual space according to this embodiment. In FIG. 5, an example in which the light source is the sun is shown in FIG. 5(a), and an example in which the light source is an indoor light is shown in FIG. 5(b). In the case of FIG. 5(a) in which the sun is assumed as the light source in global coordinates 501, which are coordinates of the virtual space, the light source is a parallel light source, and the direction of the light source (sun) is defined by the light source direction 502. Furthermore, the light source information may include other light source information 503 such as light intensity (total luminous flux and illuminance), light color (color temperature), and the ratio of ambient light, which is light that is uniformly incident from all directions, to the entire light source.

[0048] Next, an example of light source information in the case of indoor lighting as shown in Fig. 5(b) will be described. When indoor lighting is used, the light source information may include a light source position 504 indicating the position (spatial coordinates) of the light source in global coordinates 501, a light source direction 505 indicating the direction of the light source, and a light source spot angle 506 indicating the light distribution angle of the spot light when the light source is a spot light.

[0049] Note that a single virtual space may have multiple light sources, in which case light source information corresponding to each light source is stored. Furthermore, if the light source moves over time, or if changes occur in the light source, such as changes in light intensity or color, the light source information may be stored as timed metadata so that it can be handled as dynamic information with a time axis.

[0050] Next, Fig. 6 is a schematic diagram showing an example of visual field information of a virtual space according to this embodiment. In Fig. 6, visual field information is shown by a visual field position 602 indicating the spatial coordinates of a visual field (camera) in a global coordinate 601, which is a coordinate of a virtual space, a visual field direction 603 indicating the direction of a visual line from the visual field position 602, a visual field angle 604 indicating the angle of view when looking in the direction indicated by the visual field direction 603, and a rotation angle 605 indicating the rotation angle of the visual field angle 604 from a horizontal state. Note that the visual field position 602, the visual field direction 603, the visual field angle 604, and the rotation angle 605 may change over time, so that these pieces of information may be stored as timed metadata so that they can be handled as dynamic information having a time axis. In addition, as visual field information, information indicating whether the visual field is stereo or not may be stored, and a plurality of pieces of visual field information may be stored so that the visual field can be selected.

[0051] The parameters of the various information constituting the above-mentioned field of view information will now be described. The viewpoint position 602 can be represented by parameters of three axes (X, Y, Z) in Cartesian coordinates. The line of sight direction 603 is vector information from the viewpoint position 602, and therefore can be represented by three-axis parameters (X, Y, Z) with the viewpoint position 602 as the reference position. In the case of a rectangular area as shown in Fig. 6, the field of view angle 604 can be represented by angles in the horizontal and vertical directions. The rotation angle 605 can be represented as a clockwise or counterclockwise angle as viewed from the viewpoint side, based on a state in which the horizontal side of the rectangular area represented by the field of view angle 604 is parallel to the XY plane.

[0052] In this manner, in this embodiment, in addition to the decoding information, light source information or field of view information can be stored as common metadata. Here, when decoding information, which is static information, is stored in an ISOBMFF file, it is advisable to describe it in a sample description box (SampleDescriptionBox) in a base track. Note that a "sample" here is a set of data associated with a predetermined unit time of video, audio, or 3D object stored in an ISOBMFF file. As a sample, for example, a set of data to be reproduced and processed in a unit time is used, which is represented by a frame rate in the case of video, a sampling rate in the case of audio, or the like.

[0053] Regarding light source information or field of view information, if it is static data, it may be stored in a sample description box like the decoding information, but since this is not information about encoded data, it may be stored in a meta box (MetaBox) in the base track. On the other hand, if light source information or field of view information is dynamic data, it is preferable to store it as timed metadata managed in the base track.

[0054] In this embodiment, the form in which 3D coordinate transformation information is stored in the base track or each geometry track has been described with reference to Fig. 3. However, when the 3D coordinate transformation information is static information, it may also be stored in a meta box. In this case, the geometry track and the 3D coordinate transformation information can be associated with each other by defining reference information 311 to a meta box in which the 3D coordinate transformation information is stored in the geometry track as shown in Fig. 3.

[0055] In this way, the location within the file where various information is stored is not particularly limited, and implementation can be in any form as long as multiple objects can be defined in the same way within one file.

[0056] According to this configuration, it is possible to generate, for multiple three-dimensional objects, placement information for placing them in the same coordinate system and decoding information indicating whether the objects can be decoded, and to store the generated information in a single file.

[0057] The information processing device 100 according to this embodiment generates a file storing multiple pieces of 3D object data. In this embodiment, the playback device 700 performs playback processing of the 3D object data from the file generated by the information processing device 100. The playback device 700 according to this embodiment will be described below with reference to Figs. 7 and 8. Fig. 7 is a block diagram showing an example of the functional configuration of the playback device 700 according to this embodiment. The playback device 700 includes a configuration analysis unit 701, a metadata extraction unit 702, a conversion information extraction unit 703, a data extraction unit 704, a data decoding unit 705, and a rendering unit 706.

[0058] The configuration analysis unit 701 analyzes the file storing the 3D object data and identifies the locations of various data. Here, the configuration analysis unit 701 can identify the locations of the common metadata and the 3D object data from the file generated by the information processing device 100.

[0059] The metadata extraction unit 702 analyzes the decoding information in the file and determines whether the encoded 3D object data can be decoded. Specifically, the metadata extraction unit 702 analyzes the common metadata of the file and determines whether the 3D object data can be decoded based on the profile information.

[0060] The conversion information extraction unit 703 extracts 3D coordinate conversion information, which is coordinate conversion information for arranging the stored 3D object data in a virtual space.

[0061] The data extraction unit 704 extracts encoded data related to 3D object data such as a video or an image from a file. The encoded data is stored in the mdat 310 described with reference to Fig. 3. From this encoded data, it is possible to extract geometry data, which is actual data obtained by encoding coordinate information constituting the 3D object data, and attribute data, which is actual data obtained by encoding attribute information.

[0062] The data decoding unit 705 decodes the encoded geometry data and attribute data based on the determination result of the metadata extraction unit 702 .

[0063] The rendering unit 706 has a function of rendering 3D object data at predetermined coordinates in the virtual space based on the 3D coordinate conversion information extracted by the conversion information extraction unit 703. Hereinafter, rendering 3D object data stored in a file in the virtual space in this manner will be referred to as "playing".

[0064] Note that field of view information is required for rendering 3D object data. Therefore, for example, at least a part of the various types of field of view information (viewpoint position 602, line of sight direction 603, field of view angle 604, and rotation angle 605) described with reference to Fig. 6 may be stored in advance in a file as common metadata. In this case, the metadata extraction unit 702 can identify the 3D object data and the area to be rendered by reading the field of view information from the common metadata.

[0065] Next, a series of processes from acquisition of a 3D object data storage file to rendering of 3D object data by the playback device according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of playback processing by the playback device according to this embodiment.

[0066] In S801, the configuration analysis unit 701 acquires a file storing encoded 3D object data. In S802, the configuration analysis unit 701 analyzes the acquired 3D object data storage file.

[0067] In S803, the metadata extraction unit 702 extracts profile information stored as a common parameter of the 3D object data. In S804, the metadata extraction unit 702 analyzes the profile information and determines whether the 3D object data stored in the file can be decoded. If the 3D object data can be decoded, the process proceeds to S805, and if not, the process ends.

[0068] In S805, the conversion information extraction unit 703 extracts 3D coordinate conversion information from the file. In S806, the data extraction unit 704 extracts geometry data and attribute data constituting the 3D object data from the file. In S807, the data decoding unit 705 decodes the extracted 3D object data.

[0069] In S808, the rendering unit 706 uses the extracted 3D coordinate transformation information to determine position information in the virtual space of the decoded 3D object data, and performs rendering using predetermined field of view information. At this time, the field of view information may be included in the common parameters as described above, or may be held by the playback device 700.

[0070] [Variations] 3 shows an example of a file structure in which a base track 302 for managing common metadata is prepared in addition to tracks for managing coordinate information and attribute information. However, the file structure is not limited to this, and may be, for example, a form in which common metadata is stored in a geometry track. Below, a modified example of such a file structure will be described with reference to FIG.

[0071] 9 includes boxes ftyp, moov 901, and mdat 909. The basic structure and function of the file 900 are similar to those of the file 300, so a duplicated description will be omitted.

[0072] 3, moov901 includes geometry tracks 1-4 and attribute tracks 1-4 corresponding to objects 1-4. Unlike file 300, file 900 does not include a base track, and the information stored in base track 302 is stored in geometry track 1 (902). That is, geometry track 1 (902) in file 900 stores, in addition to the information stored in geometry track 306, common metadata and 3D coordinate transformation information stored in base track 302 in file 300.

[0073] Generally, media stored in a format based on ISOBMFF can be arbitrarily selected by a playback device to play back the track. Therefore, the file 900 may be generated so that the content required for playing the file (e.g., background content) is stored in the track that necessarily undergoes analysis (here, the geometry track 902).

[0074] [Embodiment 2] In the above-described embodiments, each processing unit shown in, for example, FIG. 1 is realized by dedicated hardware. Some or all of the processing units included in the information processing device 100 may be realized by a computer. In this embodiment, at least a part of the processing according to each of the above-described embodiments is executed by a computer.

[0075] FIG. 10 is a diagram showing the basic configuration of a computer. In FIG. 10, a processor 1001 is, for example, a CPU, and controls the operation of the entire computer. A memory 1002 is, for example, a RAM, and temporarily stores programs, data, etc. A computer-readable storage medium 1003 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. for the long term. In this embodiment, a program that realizes the functions of each unit, which is stored in the storage medium 1003, is read into the memory 1002. Then, the processor 1001 operates according to the program on the memory 1002, thereby realizing the functions of each unit.

[0076] In Fig. 10, an input interface 1004 is an interface for acquiring information from an external device. An output interface 1005 is an interface for outputting information to an external device. A bus 1006 connects the above-mentioned units and enables data exchange. The playback device 700 can also be implemented by hardware similar to that shown in Fig. 10.

[0077] The disclosure of this specification includes the following information processing device, information processing method, and program.

[0078] (Item 1) an acquisition means for acquiring first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object; a first generating means for generating placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; a second generating means for generating metadata common to the first three-dimensional object and the second three-dimensional object, the metadata including information related to decoding the first three-dimensional object and the second three-dimensional object; a third generating means for generating a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; a fourth generating means for generating a single file that stores the first track, the second track, and the third track, the first data, the second data, the arrangement information, and the metadata; An information processing device comprising: (Item 2) The information processing device described in item 1, characterized in that the placement information includes information indicating the position in the same coordinate system, information indicating the orientation in the same coordinate system, and information indicating the size in the same coordinate system of each of the first and second three-dimensional objects. (Item 3) The information processing device described in item 2, characterized in that the information indicating the positions of each of the first and second three-dimensional objects in the same coordinate system is a vector between the coordinates of the origin of the local coordinate system of each of the first and second three-dimensional objects and the coordinates of the origin of the same coordinate system, evaluated in a three-dimensional Cartesian coordinate system. (Item 4) The information processing device described in item 2 or 3, characterized in that information indicating the orientation of each of the first and second three-dimensional objects in the same coordinate system is a rotation angle between a coordinate axis of each of the local coordinate systems of the first and second three-dimensional objects and a coordinate axis of the same coordinate system. (Item 5) The information processing device according to any one of items 2 to 4, characterized in that the information indicating the size of each of the first and second three-dimensional objects in the same coordinate system is an enlargement or reduction ratio for each of the X-axis, Y-axis, and Z-axis values ​​between the local coordinate systems of each of the first and second three-dimensional objects and the same coordinate system, evaluated in a three-dimensional Cartesian coordinate system. (Item 6) 6. The information processing device according to any one of items 1 to 5, wherein the origin of the identical coordinate system is a reference position of a space managed by a space ID. (Item 7) 7. The information processing device according to any one of items 1 to 6, wherein the metadata is a parameter set that is referenced when decoding the first and second three-dimensional objects. (Item 8) the third generating means generates boxes for storing index information indicating a reference destination of data as the first, second, and third tracks; The information processing device described in any one of items 1 to 7, characterized in that the fourth generation means generates the single file so as to manage the placement information and the metadata as an index using the same box. (Item 9) The second generating means further generates visual field information indicating a viewpoint in the same coordinate system, Item 9. The information processing device according to item 8, wherein the fourth generating means generates the single file so as to manage the metadata and the field of view information as indexes using the same box. (Item 10) 10. The information processing device according to item 9, wherein the field of view information is information indicating a viewpoint position in the same coordinate system. (Item 11) The second generating means further generates light source information indicating a light source in the same coordinate system, The information processing device described in any one of items 8 to 10, characterized in that the third generation means generates the single file so as to manage the metadata and the light source information as indexes using the same box. (Item 12) Item 12. The information processing device according to item 11, wherein the light source information includes at least one of the position, direction, light intensity, and light color of the light source in the same coordinate system. (Item 13) 13. The information processing device according to item 11 or 12, wherein the light source information includes, when the light source includes a spot light, a light distribution angle of the spot light. (Item 14) 14. The information processing device according to any one of items 11 to 13, wherein the light source information is information indicating a plurality of light sources in the same coordinate system. (Item 15) The information processing device described in any one of items 11 to 14, characterized in that the light source information includes information indicating the ratio of the ambient light to the entire light source when the light source includes ambient light that is uniformly incident from all directions. (Item 16) An acquisition means for acquiring a single file storing data of a first three-dimensional object and data of a second three-dimensional object; a first acquiring means for acquiring, from the file, placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; a second obtaining means for obtaining from the file metadata common to the first three-dimensional object and the second three-dimensional object, the metadata including information related to decoding the first three-dimensional object and the second three-dimensional object; a decoding means for decoding data of the first three-dimensional object and data of the second three-dimensional object when the first three-dimensional object and the second three-dimensional object are decodable; a rendering means for rendering the first three-dimensional object and the second three-dimensional object in a space of the same coordinate system based on the placement information and the first three-dimensional object and the second three-dimensional object decoded by the decoding means; A playback device comprising: (Item 17) The playback device described in item 16, characterized in that the placement information includes information indicating the position in the same coordinate system, information indicating the orientation in the same coordinate system, and information indicating the size in the same coordinate system of each of the first and second three-dimensional objects. (Item 18) 18. The playback device according to item 16 or 17, characterized in that the origin of the identical coordinate system is a reference position of a space managed by a space ID. (Item 19) 19. A playback device according to any one of items 16 to 18, characterized in that the metadata is profile information that is referenced when decoding the first and second three-dimensional objects. (Item 20) The second acquisition means further acquires, from the file, visual field information indicating a viewpoint in the same coordinate system; 20. The playback device according to any one of items 16 to 19, wherein the rendering means renders the first three-dimensional object and the second three-dimensional object based on the field of view information. (Item 21) The second acquisition means further acquires light source information indicating a light source in the same coordinate system from the file, 21. The playback device according to any one of items 16 to 20, wherein the rendering means renders the first three-dimensional object and the second three-dimensional object based on the light source information. (Item 22) obtaining first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object; generating placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; generating common metadata for the first and second three-dimensional objects, the metadata comprising information related to decoding the first and second three-dimensional objects; generating a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; generating a single file that stores the first track, the second track, and the third track, the first data, the second data, the arrangement information, and the metadata; An information processing method comprising: (Item 23) obtaining a single file storing data of the first three-dimensional object and data of the second three-dimensional object; acquiring, from the file, placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; obtaining from the file metadata common to the first and second three-dimensional objects, the metadata comprising information related to decoding the first and second three-dimensional objects; if the first and second three-dimensional objects are decodable, decoding the data of the first and second three-dimensional objects; rendering the first three-dimensional object and the second three-dimensional object in the space of the same coordinate system based on the placement information and the decoded first three-dimensional object and the second three-dimensional object; An information processing method comprising: (Item 24) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 15 or the playback device according to any one of items 16 to 21.

[0079] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0080] For example, the present invention can be embodied as a system, an apparatus, a method, a program, a recording medium (storage medium), etc. Specifically, the present invention may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device.

[0081] The present invention can also be achieved by directly or remotely supplying a software program for implementing the functions of the information processing device in the above-described embodiment to a system or device, and having the computer in the system or device read and execute the supplied program code. Note that the program in this case is a computer-readable program that executes the processing corresponding to the flowchart shown in the figure in the first embodiment.

[0082] Therefore, the program code itself installed in a computer to realize the functional processing by the information processing device of the present invention also realizes the present invention. In other words, the present invention also includes the computer program itself for realizing the functional processing by the information processing device of the present invention. In this case, as long as it has the function of the program, it may be in the form of object code, a program executed by an interpreter, script data supplied to an OS, or the like.

[0083] The following media are available as recording media for supplying the program: floppy disks, hard disks, optical disks, magneto-optical disks, MOs, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, non-volatile memory cards, ROMs, and DVDs (DVD-ROMs or DVD-Rs).

[0084] The following methods are also possible for supplying the program. That is, the browser of the client computer may connect to an Internet homepage, from which the computer program of the present invention itself (or a compressed file including an automatic installation function) may be downloaded to a recording medium such as a hard disk. It may also be realized by dividing the program code constituting the program of the present invention into multiple files, and downloading each file from a different homepage. In other words, the present invention also includes a WWW server that allows multiple users to download program files for implementing the functional processing of the present invention on a computer.

[0085] It is also possible to encrypt the program of the present invention, store it on a storage medium such as a CD-ROM, and distribute it to users, and allow users who meet certain conditions to download key information for decrypting the program from a website via the Internet. The users can use the key information to install the encrypted program on their computers.

[0086] Moreover, the functions of the above-mentioned embodiment are realized by the computer executing the read program. Furthermore, based on the instructions of the program, the OS running on the computer performs part or all of the actual processing, and the functions of the above-mentioned information processing device can also be realized by this processing. Furthermore, the functions of the above-mentioned information processing device can also be realized by writing the program read from the recording medium to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, and then executing the program. In other words, based on the instructions of the program, the CPU provided in the function expansion board or function expansion unit can perform part or all of the actual processing.

[0087] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0088] 100: information processing device, 101: data acquisition unit, 102: conversion information generation unit, 103: data analysis unit, 104: data generation unit, 105: track generation unit, 106: file storage unit

Claims

1. an acquiring means for acquiring first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object; a first generating means for generating placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; - a second generating means for generating metadata common to the first and second three-dimensional objects, the metadata including information related to decoding the first and second three-dimensional objects; a third generating means for generating a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; a fourth generating means for generating a single file that stores the first track, the second track, and the third track, the first data, the second data, the arrangement information, and the metadata; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the placement information includes information indicating a position in the same coordinate system, information indicating a posture in the same coordinate system, and information indicating a size in the same coordinate system of each of the first and second three-dimensional objects.

3. 3. The information processing device according to claim 2, characterized in that the information indicating the positions of each of the first and second three-dimensional objects in the same coordinate system is a vector between the coordinates of the origin of the local coordinate system of each of the first and second three-dimensional objects and the coordinates of the origin of the same coordinate system, evaluated in a three-dimensional Cartesian coordinate system.

4. 3. The information processing device according to claim 2, wherein information indicating the orientation of each of the first and second three-dimensional objects in the same coordinate system is a rotation angle between a coordinate axis of each of the local coordinate systems of the first and second three-dimensional objects and a coordinate axis of the same coordinate system.

5. 3. The information processing device according to claim 2, wherein the information indicating the size of each of the first and second three-dimensional objects in the same coordinate system is an enlargement or reduction ratio for each of the X-axis, Y-axis, and Z-axis values ​​between the local coordinate systems of each of the first and second three-dimensional objects and the same coordinate system, evaluated in a three-dimensional Cartesian coordinate system.

6. The information processing apparatus according to claim 1 , wherein the origin of said uniform coordinate system is a reference position of a space managed by a space ID.

7. The information processing apparatus according to claim 1 , wherein the metadata is a parameter set that is referred to when the first and second three-dimensional objects are decoded.

8. the third generating means generates boxes for storing index information indicating a reference destination of data as the first, second, and third tracks; The information processing apparatus according to claim 1 , wherein the fourth generating means generates the single file so that an index for the arrangement information and the metadata is managed by the same box.

9. The second generating means further generates visual field information indicating a viewpoint in the same coordinate system, 9. The information processing apparatus according to claim 8, wherein the fourth generating means generates the single file so that an index of the metadata and the visual field information is managed by the same box.

10. The information processing apparatus according to claim 9 , wherein the visual field information is information indicating a viewpoint position in the same coordinate system.

11. The second generating means further generates light source information indicating a light source in the same coordinate system, 9. The information processing apparatus according to claim 8, wherein the fourth generating means generates the single file so that indexes of the metadata and the light source information are managed by the same box.

12. 12. The information processing apparatus according to claim 11, wherein the light source information includes at least one of a position, a direction, a light intensity, and a light color of the light source in the same coordinate system.

13. The information processing apparatus according to claim 11 , wherein the light source information includes, when the light source includes a spot light, a light distribution angle of the spot light.

14. The information processing apparatus according to claim 11 , wherein the light source information is information indicating a plurality of light sources in the same coordinate system.

15. 12. The information processing apparatus according to claim 11, wherein the light source information includes information indicating a ratio of the ambient light to an entire light source when the light source includes ambient light uniformly incident from all directions.

16. An acquisition means for acquiring a single file storing data of a first three-dimensional object and data of a second three-dimensional object; a first acquiring means for acquiring, from the file, placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; a second obtaining means for obtaining metadata common to the first three-dimensional object and the second three-dimensional object from the file, the metadata including information related to decoding the first three-dimensional object and the second three-dimensional object; a decoding means for decoding data of the first three-dimensional object and data of the second three-dimensional object when the first three-dimensional object and the second three-dimensional object are decodable; a rendering means for rendering the first three-dimensional object and the second three-dimensional object in a space of the same coordinate system based on the placement information and the first three-dimensional object and the second three-dimensional object decoded by the decoding means; A playback device comprising:

17. 17. The playback device of claim 16, wherein the placement information includes information indicating a position in the same coordinate system, information indicating a posture in the same coordinate system, and information indicating a size in the same coordinate system of each of the first and second three-dimensional objects.

18. 17. The playback device according to claim 16, wherein the origin of the identical coordinate system is a reference position of a space managed by a space ID.

19. 17. The playback device according to claim 16, wherein the metadata is profile information that is referenced when the first and second three-dimensional objects are decoded.

20. The second acquisition means further acquires, from the file, visual field information indicating a viewpoint in the same coordinate system; 17. The playback device according to claim 16, wherein the rendering means renders the first three-dimensional object and the second three-dimensional object based on the field of view information.

21. The second acquisition means further acquires light source information indicating a light source in the same coordinate system from the file, 17. The playback device according to claim 16, wherein the rendering means renders the first three-dimensional object and the second three-dimensional object based on the light source information.

22. obtaining first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object; generating placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; generating common metadata for the first and second three-dimensional objects, the metadata comprising information related to decoding of the first and second three-dimensional objects; generating a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; generating a single file that stores the first track, the second track, and the third track, the first data, the second data, the arrangement information, and the metadata; An information processing method comprising:

23. obtaining a single file storing data of the first three-dimensional object and data of the second three-dimensional object; acquiring, from the file, placement information for placing the first three-dimensional object and the second three-dimensional object in the same coordinate system; obtaining from the file metadata common to the first and second three-dimensional objects, the metadata comprising information related to decoding the first and second three-dimensional objects; if the first and second three-dimensional objects are decodable, decoding the data of the first and second three-dimensional objects; rendering the first three-dimensional object and the second three-dimensional object in the space of the same coordinate system based on the placement information and the decoded first three-dimensional object and the second three-dimensional object; An information processing method comprising:

24. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 15 or the reproducing device according to any one of claims 16 to 21.

Images