Information processing device, playback device, information processing method, and program

Abstract

The obtaining section obtains first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object. The first generation means generates arrangement information for arranging the first three-dimensional object and the second three-dimensional object in the same coordinate system. The second generation section generates metadata common to the first three-dimensional object and the second three-dimensional object. A third generation section generates a first track that manages the first data, a second track that manages the second data, and a third track that manages the metadata. A fourth generation section generates a single file storing the first rail, the second rail, the third rail, the first data, the second data, the arrangement information, and the metadata.

Description

Technical Field

[0001] This invention relates to an information processing device, a playback device, an information processing method, and a program. Background Technology

[0002] Computer graphics are known as a traditional method for generating 3D object data. However, in recent years, methods for obtaining 3D object data by scanning the shapes of actual objects and people using specialized equipment and studios are becoming more frequently used.

[0003] Efforts are also underway to utilize data that arranges multiple 3D object data items generated or obtained in this way within the same three-dimensional space. For example, in the field of autonomous driving or driver assistance, progress is being made in the development of systems that use vehicle-mounted remote sensing devices, such as LiDAR (Light Detection and Ranging) or Laser Imaging Detection and Ranging, to acquire 3D object data of objects around the road and overlay this 3D object data onto a dynamic map to update road information in real time.

[0004] On the other hand, MPEG (Moving Picture Experts Group), a group under ISO (International Organization for Standardization) and IEC (International Electrotechnical Commission), is promoting the standardization of specifications for encoding 3D object data such as point clouds or meshes, as well as file format standards for storing encoded 3D object data.

[0005] Patent document 1 discloses a technique that enables the quality of a portion of 3D object data to be altered, for example, by spatially segmenting 3D object data encoded via G-PCC (geometry-based point cloud compression) being standardized by MPEG, and then generating positional information relating to the position of each part of the segmented 3D object data in three-dimensional space and grouping information indicating that the segmented parts belong to the same group. Existing technical documents Patent documents

[0006] Patent Document 1: International Publication No. 2020 / 137642 Summary of the Invention The problem the invention aims to solve

[0007] However, although the technology described in Patent Document 1 discloses aspects of segmenting a single 3D object data item, it does not take into account aspects of storing multiple different 3D object data items in a single file.

[0008] Furthermore, glTF, as a format for representing 3D models, uses JSON to write the structure and configuration of the 3D space, allowing multiple 3D object data items to be arranged in the same 3D space. However, glTF is a specification that uses URIs to associate 3D space and object data with each other, and it already requires reading and analyzing each object data item to determine whether the object data is in a displayable data format.

[0009] The purpose of this invention is to utilize a single file that stores information for displaying multiple three-dimensional object data items in the same coordinate system. Solution for solving the problem

[0010] To achieve the objectives of the present invention, for example, an information processing apparatus according to one embodiment has the following configuration. In other words, an information processing apparatus includes: an obtaining component for obtaining first data of a first three-dimensional object and second data of a second three-dimensional object different from the first three-dimensional object; a first generating component for generating arrangement information for arranging the first three-dimensional object and the second three-dimensional object in the same coordinate system; a second generating component for generating metadata common to the first three-dimensional object and the second three-dimensional object; a third generating component for generating a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; and a fourth generating component for generating a single file storing the first track, the second track, the third track, the first data, the second data, the arrangement information, and the metadata. Advantages of the invention

[0011] It utilizes a single file that stores information used to display multiple 3D object data items in the same coordinate system.

[0012] Other features and advantages of the invention will become apparent from the following description taken in conjunction with the accompanying drawings. Note that throughout the drawings, the same reference numerals denote the same or similar components. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the specification, serve to explain the principles of the invention. Figure 1 This is a block diagram illustrating an example of the functional configuration of an information processing device according to a first embodiment. Figure 2 This is a flowchart illustrating an example of a file generation process according to the first embodiment. Figure 3 This is a diagram illustrating the structure of a file generated by the information processing device according to the first embodiment. Figure 4 This is a diagram illustrating the arrangement information according to the first embodiment. Figure 5A and Figure 5B This is a diagram illustrating light source information according to the first embodiment. Figure 6 This is a diagram illustrating the field of view information according to the first embodiment. Figure 7 This is a block diagram illustrating an example of the functional configuration of a playback device according to the first embodiment. Figure 8 This is a flowchart illustrating an example of the rendering process according to the first embodiment. Figure 9 This is a diagram showing the structure of a file generated in a variant of an information processing device. Figure 10 This is a block diagram illustrating an example of the hardware configuration of a device according to a second embodiment. Detailed Implementation

[0014] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but this does not limit the invention to requiring all such features, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0015] [First Embodiment] Reference Figure 1 The functional configuration of an information processing device according to an embodiment of the present invention is described. Figure 1 This is a block diagram illustrating the functional configuration of the information processing device 100 according to the first embodiment. The information processing device 100 according to this embodiment includes a data acquisition unit 101, a conversion information generation unit 102, a data analysis unit 103, a data generation unit 104, a track generation unit 105, and a file storage unit 106.

[0016] The data acquisition unit 101 acquires encoded three-dimensional (3D) object data, such as encoded point cloud data or 3D mesh data. The data acquisition unit 101 can acquire various types of data from, for example, an external device (not shown). In this embodiment, the 3D object data acquired by the data acquisition unit 101 is encoded, and the 3D object data includes information used when decoding the 3D object data (decoding information). According to this embodiment, the decoding information includes metadata related to decoding and includes a set of parameters (profile information) referenced by the decoder during decoding. In this embodiment, the "decoding-related information" includes information indicating whether the 3D object data can be decoded as a parameter set, and a determination of whether decoding is possible is made by analyzing this parameter set.

[0017] Here, the decoding information is information referenced during decoding (such as identifying the level or grade of the toolset used during encoding), which can be added using known encoding techniques; therefore, its detailed description will be omitted. The data analysis unit 103 extracts the decoding information by analyzing the obtained 3D object data. In the following text, the 3D object represented by such 3D object data can be simply referred to as an "object".

[0018] The transformation information generation unit 102 generates arrangement information for arranging the obtained 3D object data in a given three-dimensional space. According to this embodiment, the arrangement information is 3D coordinate transformation information used to convert the local coordinates of two or more 3D objects into coordinates in the same coordinate system; this will be referred to later. Figure 3 and Figure 4 A detailed description of it will be given below. In this embodiment, local coordinates of each 3D object are generated from the corresponding 3D object data obtained from the data acquisition unit 101, and then these local coordinates are converted into global coordinates in the same coordinate system.

[0019] The data generation unit 104 generates metadata for managing the decoded information extracted by the data analysis unit 103. In this embodiment, under the assumption that the obtained decoded information for multiple 3D object data items is common, it provides methods for generating metadata such as... Figure 3 The public metadata, such as the public metadata indicated by 303 (described later), refers to the functionality of public metadata. According to this embodiment, the public metadata includes public information and decoding information used to draw each object. It is assumed that this public information is, for example, light source information or field of view information, and will be referred to later... Figure 5A , Figure 5B and Figure 6 Let me describe this in detail.

[0020] According to this embodiment, the track generation unit 105 generates trak blocks (tracks; described later) in the file generated by the information processing device 100 as blocks for managing various types of data. Here, the track generation unit 105 can generate tracks for storing information related to the obtained 3D object data and tracks for managing public metadata generated by the data generation unit 104.

[0021] File storage unit 106 generates and stores multiple 3D object data items obtained from storage and files of tracks generated by track generation unit 105.

[0022] Next, we will refer to Figure 2 The process sequence from obtaining 3D object data to generating a file is described by the information processing device 100 according to this embodiment. Figure 2 This is a flowchart illustrating an example of a file generation process performed by an information processing device according to this embodiment. For example, when 3D object data is sent from an external device (not shown) to the information processing device 100, the process begins according to... Figure 2 The processing.

[0023] In S201, the data acquisition unit 101 acquires 3D object data. Here, the data acquisition unit 101 acquires 3D object data from an external device. In S202, the track generation unit 105 generates tracks for managing the acquired 3D object data. (See below for further details.) Figure 3 Describes the track used to manage 3D object data.

[0024] In S203, the data acquisition unit 101 determines whether all object data to be stored in the file has been acquired in S201. If all object data has been acquired, the sequence moves to S204; otherwise, the sequence returns to S201. Here, the data acquisition unit 101 is configured to acquire, for example, a predetermined number of 3D object data items, and can determine whether the predetermined number of 3D object data items has been acquired. For example, the data acquisition unit 101 can make this determination by receiving a signal from an external device that transmits the 3D object data, indicating whether all 3D object data has been transmitted.

[0025] In S204, the transformation information generation unit 102 sets the origin (reference position) in a coordinate system (global coordinate system) of a given three-dimensional space in which the acquired 3D object data is to be arranged. This global coordinate system is used when displaying each 3D object within a 3D object and can be set as needed. For example, the global coordinate system can be a coordinate system based on geographic coordinates specified by GPS (Global Positioning System). The global coordinate system can also be a coordinate system used to display the coordinates of each space within a space defined by a space ID. In this case, it is assumed that a predetermined position in the space assigned by the space ID (e.g., one of the angles of space such as a voxel assigned by the space ID) in the global coordinate system is represented as the reference point of that space ID.

[0026] According to this embodiment, a spatial ID is identification information that specifies a spatial location within the same global coordinate system. In other words, in this embodiment, a spatial ID can uniquely specify the location of a given 3D space in the real world, and thus the positional relationship between spaces specified by multiple spatial IDs can also be uniquely specified. Therefore, when using a spatial ID as a reference location, reference points with different spatial IDs for each 3D object data item can be set as reference locations.

[0027] Furthermore, although this embodiment assumes that the transformation results in all 3D objects being processed being arranged in the same global coordinate system, this embodiment is not particularly limited to this. For example, 3D objects can be divided into several groups, and the local coordinates of each 3D object can be transformed into coordinates for a different three-dimensional space of each group.

[0028] This embodiment assumes that spatial positions in a common global coordinate system are specified by spatial IDs. However, positions in coordinate systems of different three-dimensional spaces can be specified by individual spatial IDs. For example, spatial IDs can be assigned to different levels of the structure, and a coordinate system with the angle of that level as its origin can be referenced.

[0029] In S205, the transformation information generation unit 102 generates 3D coordinate transformation information. According to this embodiment, the 3D coordinate transformation information includes information indicating the position, pose, and size of the corresponding 3D object in the global coordinate system. Here, the 3D coordinate transformation information includes, for example, the offset of the local coordinates calculated for each object relative to the origin of the global coordinate system (origin shift), the tilt of the coordinate axes relative to the global coordinate system (rotation angle), and scaling information relative to the global coordinate system (magnification / reduction ratio).

[0030] In S206, the data analysis unit 103 extracts decoding information by analyzing the obtained 3D object data. In S207, the data generation unit 104 generates public metadata, which includes decoding information used for decoding each 3D object data item.

[0031] In S208, the track generation unit 105 generates a track (basic track) that stores 3D coordinate transformation information and common parameters. In S209, the file storage unit 106 generates a file that stores the multiple 3D object data items obtained and the track generated by the track generation unit 105. Afterwards... Figure 2 The processing shown has ended.

[0032] Note that the format of the encoded 3D object data can be any format that includes geometric information as positional information in three-dimensional space, and there are no particular restrictions on the data format. For example, the 3D object data used in this embodiment can be in point cloud data format or 3D mesh data format, etc.

[0033] Next, we will refer to Figure 3 Description by Figure 2 The image shows a specific example of a file generated from the processed sequence. Figure 3 This is a schematic diagram illustrating an example of the internal structure of a file generated by the information processing device 100 according to this embodiment. Although Figure 3 The file format shown will be described as being based on the ISO Basic Media File Format (hereinafter referred to as ISOBMFF), but the file format is not particularly limited to this, as long as data can be stored in the same way, wherein the ISO Basic Media File Format is the basic specification for media files standardized by MPEG.

[0034] The file 300 generated by the information processing device 100 according to this embodiment includes a plurality of boxes identified by four-character identifiers, and each of these boxes stores information for various purposes. Hereinafter, each box will be represented by a four-character identifier assigned to it. The file 300 according to this embodiment includes ftyp, moov 301, and mdat 310 as boxes. ftyp (FileTypeBox) has a four-character identifier called brand for identifying the type / subtype of the image file.

[0035] The moov (MovieBox) 301 includes multiple track boxes. According to this embodiment, the moov 301 can store metadata related to data such as motion pictures or images, and the track boxes (tracks) can store data indicating the arrangement position (index information) of data such as motion pictures, images, or audio. Here, the moov 301 includes a total of nine tracks, including a basic track 302 and tracks 306 to 309. Here, the moov 301 stores geometric tracks 1 to 4 that manage coordinate information representing the shape of an object, and attribute tracks 1 to 4 that manage attribute information of the object (such as the color of the object's surface or the reflectivity of light).

[0036] exist Figure 3 In the example, moov 301 stores information corresponding to four independent objects 1 to 4, and this information is indicated by geometry tracks 1 to 4 and attribute tracks 1 to 4, respectively. In other words, geometry track 1 (306) and attribute track 1 (307) are included as blocks storing information corresponding to, for example, object 1. Note that the association between the geometry track and the attribute track is defined by generating reference information to the attribute track in the geometry track.

[0037] Note that the actual data for the coordinate and attribute information managed by these tracks is stored in mdat (MediaDataBox) 310. In other words, the information stored in moov 301 is index information indicating the reference location of the actual data stored in mdat 310, and although for ease of interpretation... Figure 3 The mdat 310 is shown in small size, but the corresponding data stored in mdat 310 is much larger than the information stored in the corresponding track.

[0038] The basic track 302 stores public metadata 303 and 3D coordinate transformation information corresponding to each object. Figure 3 In the example, the 3D coordinate transformation information 1 to 4 corresponding to objects 1 to 4 is information used to arrange the object data at global coordinates in a common three-dimensional space (hereinafter referred to as virtual space). Here, local coordinates are managed as coordinate information in the geometric orbit of each object. Therefore, as mentioned above, the 3D coordinate transformation information includes the offset of the local coordinates relative to the origin of the global coordinate system (the amount of origin shift), the tilt of the coordinate axes relative to the global coordinate system (rotation angle), and the scaling information (magnification / reduction ratio) relative to the global coordinate system for each object.

[0039] Here, we will refer to Figure 4 A specific example describing 3D coordinate transformation information. Figure 4This is a schematic diagram illustrating an example of 3D coordinate transformation information according to this embodiment. Figure 4 In this context, global coordinate 401 is a single set of spatial coordinates in the virtual space, and local coordinates 1 to 4 are unit vectors used in the coordinate systems representing the local coordinates of objects 1 to 4. Local coordinate 1 (402) will be described below.

[0040] Origin offset 403 is information indicating the position of the origin of local coordinate 1 in virtual space relative to the origin of global coordinate 401 as an offset value, and can be used in... Figure 4 The three-dimensional orthogonal coordinate system shown is indicated by the offset values ​​(dx, dy, dz) of the three axes.

[0041] Additionally, Figure 4 The ΔX(404), ΔY(405), and ΔZ(406) indicated in the text are information indicating the tilt (rotation angle) of the local coordinate axes relative to the global coordinate system 401. Figure 4 In the example, the rotation angle is indicated by the roll / pitch / yaw angle in a three-dimensional orthogonal coordinate system, but different methods can be used as long as the rotation angle can be represented, such as using quaternions.

[0042] exist Figure 4 In the example, when representing each 3D object data item in virtual space, scaling information (magnification / reduction ratio) is added to set the size of each 3D object data item to the desired ratio. The scaling information can be set for each direction, such as the X, Y, and Z directions in each local coordinate system. The value of the scaling information can be defined by integers or fractions, etc. The scaling information can be set based on user input, for example, and can be added to the 3D object data obtained by the data acquisition unit 101.

[0043] Although local coordinate 1 has been described so far, various types of 3D coordinate transformation information can be generated for other local coordinates 2 to 4 in the same way. In this way, the relative arrangement information of multiple 3D object data items in virtual space can be determined by generating 3D coordinate transformation information for each object.

[0044] With this configuration, multiple different 3D object data items can be stored in a single file, thus enabling the technology to be implemented using only one file when combining multiple 3D objects. For example, 3D object data of a background and 3D object data of a person can be combined, and a portion of a dynamic map intended for use in autonomous driving and other applications can be combined with 3D object data acquired almost in real time by LiDAR.

[0045] Note that each 3D object data item can have a different frame rate, and static 3D object data and dynamic 3D object data can even be combined with each other. Note that when storing static 3D object data in a file, consider storing the data as items in iloc and iinf boxes (not shown) instead of trak boxes.

[0046] Despite Figure 3 The 3D coordinate transformation information shown is described as being stored in the base track, but the storage location is not particularly limited to this. For example, the 3D coordinate transformation information can be stored in the track corresponding to each object (such as the geometry in geometry 1 to 4), rather than in public metadata.

[0047] Here, it is assumed that the virtual space containing multiple 3D object data items is a user-defined virtual space. However, as mentioned above, the global coordinates can be a coordinate system based on geographic coordinates specified by GPS, or a predefined coordinate system used when using spatial IDs.

[0048] Next, we will refer to Figure 5A , Figure 5B and Figure 6 Description in Figure 3 Examples of metadata other than decoding information stored in the public metadata 303 indicated in the document. Figure 5A and Figure 5B This is a schematic diagram illustrating an example of light source information in a virtual space according to this embodiment. Figure 5A and Figure 5B middle, Figure 5A An example is shown where the light source is the sun, and Figure 5B Examples are shown where the light source is indoor lighting. For example, assuming the sun is the light source. Figure 5A In the virtual space, within the global coordinates 501, the light source is a parallel light source, and the direction of the light source (the sun) is defined by the light source direction 502. Furthermore, other light source information 503 may include light intensity (total luminous flux and illumination level), light color (color temperature), and the proportion of ambient light (light incident uniformly from all directions) relative to the entire light source.

[0049] Next, it will be described as follows Figure 5B The example below illustrates light source information when the light source is indoor lighting. When using indoor lighting, light source position 504, which indicates the position of the light source in global coordinates 501 (spatial coordinates), light source direction 505, which indicates the direction of the light source, and light source point angle 506, which indicates the light distribution angle of the spotlight when the light source is a spotlight, can be used as light source information.

[0050] Multiple light sources can exist in a single virtual space. In this case, the light source information corresponding to each light source is stored. Furthermore, if the light source changes (e.g., if the light source moves over time, or if its intensity or color changes), the light source information can be stored as timed metadata so that it can be processed as dynamic information with a timeline.

[0051] Next, Figure 6 This is a schematic diagram illustrating an example of field-of-view information in a virtual space according to this embodiment. Figure 6 In this system, the field of view information is represented by viewpoint position 602, viewing direction 603, field of view angle 604, and rotation angle 605. Viewpoint position 602 indicates the spatial coordinates of the viewpoint (camera) within the global coordinates 601, which serves as the coordinates of the virtual space. Viewing direction 603 indicates the direction of view from viewpoint position 602. Field of view angle 604 indicates the viewing angle when viewed in the direction indicated by viewing direction 603. Rotation angle 605 indicates the rotation angle relative to when field of view angle 604 is horizontal. Viewpoint position 602, viewing direction 603, field of view angle 604, and rotation angle 605 may change over time; therefore, the information can be stored as temporal metadata, allowing it to be processed as dynamic information with a timeline. Additionally, information indicating whether the viewpoint is stereoscopic can be stored as field of view information, and multiple field of view information items can be stored, allowing for viewpoint selection.

[0052] Next, the parameters of the various types of information constituting the aforementioned field of view information will be described. The viewpoint position 602 can be indicated by three axis parameters (X, Y, Z) in orthogonal coordinates. The viewing direction 603 is vector information from the viewpoint position 602 and can be indicated by three axis parameters (X, Y, Z) using the viewpoint position 602 as a reference position. In situations such as... Figure 6 In the case of a rectangular region, such as the rectangular region shown, the field of view 604 can be indicated by angles in both the horizontal and vertical directions. The rotation angle 605 can be indicated as the angle to the right or left when viewed from the viewpoint side, with the horizontal side of the rectangular region indicated by the field of view 604 being parallel to the XY plane as a reference.

[0053] In this way, this embodiment assumes that, in addition to decoding information, light source information and field of view information can also be stored as public metadata. Here, if the decoding information, as static information, is stored in the ISOBMFF file, this information can be written into the SampleDescriptionBox in the base track. Here, it is assumed that a "sample" is a collection of data associated with a predetermined unit of time of video, audio, or 3D object to be stored in the ISOBMFF file. The collection of data undergoing playback processing within a unit of time is used as a sample, where this collection is represented, for example, by the frame rate in the case of video, or by the sampling rate in the case of audio.

[0054] If the light source information or field of view information is static data, it can be stored in a sample descriptor box in the same way as the decoded information; however, since this information is not related to the encoded data, it can be stored in a metabox in the base track. On the other hand, if the light source information or field of view information is dynamic data, it is desirable to store the data as temporal metadata managed by the base track.

[0055] This embodiment has been referenced. Figure 3 This describes the form in which 3D coordinate transformation information is stored in the base orbital or individual geometric orbitals. However, if the 3D coordinate transformation information is static, it can also be stored in the metabox. In this case, such as... Figure 3 As shown, the geometric track can be associated with the 3D coordinate transformation information by defining reference information 311 to the metabox that stores the 3D coordinate transformation information in the geometric track.

[0056] In this way, there are no particular restrictions on the location of files containing various types of information, and it can be implemented in any form, as long as multiple objects can be defined in the same way in a single file.

[0057] With this configuration, arrangement information for arranging multiple 3D objects in the same coordinate system and decoding information indicating whether the objects can be decoded can be generated, and the generated information can be stored in a single file.

[0058] The information processing device 100 according to this embodiment generates a file storing multiple 3D object data items. In this embodiment, the playback device 700 performs playback processing on the 3D object data based on the file generated by the information processing device 100. Hereinafter, reference will be made to... Figures 7 to 8 The playback device 700 according to this embodiment is described. Figure 7This is a block diagram illustrating an example of the functional configuration of the playback device 700 according to this embodiment. The playback device 700 includes a configuration analysis unit 701, a metadata extraction unit 702, a conversion information extraction unit 703, a data extraction unit 704, a data decoding unit 705, and a rendering unit 706.

[0059] The configuration analysis unit 701 analyzes files containing 3D object data and specifies the locations of various types of data. Here, the configuration analysis unit 701 can specify the locations of public metadata and 3D object data based on the files generated by the information processing device 100.

[0060] The metadata extraction unit 702 analyzes the decoding information in the file and determines whether the encoded 3D object data can be decoded. Specifically, it analyzes the file's public metadata and determines whether the 3D object data can be decoded based on the grade information.

[0061] The transformation information extraction unit 703 extracts 3D coordinate transformation information, which is used to arrange the stored 3D object data in virtual space.

[0062] The data extraction unit 704 extracts encoded data related to 3D object data, such as motion images or images, from the file. The encoded data is then stored in a reference... Figure 3 In the described mdat 310, geometric data and attribute data can be extracted from the encoded data. The geometric data is the encoded actual data of the coordinate information that constitutes the 3D object data, and the attribute data is the encoded actual data of the attribute information.

[0063] The data decoding unit 705 decodes the encoded geometric data and attribute data based on the judgment result of the metadata extraction unit 702.

[0064] The drawing unit 706 has the function of drawing 3D object data at predetermined coordinates in virtual space based on the 3D coordinate transformation information extracted by the transformation information extraction unit 703. In the following text, drawing 3D object data stored in a file in virtual space in this way will be referred to as "playback".

[0065] Note that drawing 3D object data requires field-of-view information. Therefore, for example, reference data can be pre-stored in a file. Figure 6 At least a portion of the various types of field-of-view information (viewpoint position 602, viewing direction 603, field of view angle 604, and rotation angle 605) are described as public metadata. In this case, the metadata extraction unit 702 can identify the 3D object data and regions to be drawn by reading the field-of-view information from the public metadata.

[0066] Next, we will refer to Figure 8 The process sequence from obtaining a file containing 3D object data to drawing 3D object data performed by the playback device according to this embodiment is described. Figure 8 This is a flowchart illustrating an example of playback processing performed by the playback device according to this embodiment.

[0067] In S801, the configuration analysis unit 701 obtains a file containing the encoded 3D object data. In S802, the configuration analysis unit 701 analyzes the obtained file containing the 3D object data.

[0068] In S803, the metadata extraction unit 702 extracts the grade information stored as a common parameter of the 3D object data. In S804, the metadata extraction unit 702 analyzes the grade information and determines whether the 3D object data stored in the file can be decoded. If the data can be decoded, the sequence moves to S805; otherwise, the sequence ends.

[0069] In S805, the transformation information extraction unit 703 extracts 3D coordinate transformation information from the file. In S806, the data extraction unit 704 extracts the geometric data and attribute data constituting the 3D object data from the file. In S807, the data decoding unit 705 decodes the extracted 3D object data.

[0070] In S808, using the extracted 3D coordinate transformation information, the drawing unit 706 determines the position information of the decoded 3D object data in the virtual space, and uses predetermined field of view information to draw the 3D object data. At this time, the field of view information can be included in the common parameters described above, or the playback device 700 can have field of view information.

[0071] [Variation Example] Figure 3 The following is an example of a file structure in which the basic track 302, used to manage public metadata, is provided separately from the track used to manage coordinate and attribute information. However, the file structure is not limited to this, and for example, public metadata can be stored in the geometric track. References will follow below. Figure 9 This describes a variation of the file structure.

[0072] Figure 9 The document 900 shown includes ftyp, moov 901, and mdat 909 as boxes. The basic structure and function of document 900 are the same as those of document 300, therefore redundant descriptions will be omitted.

[0073] and Figure 3Similarly, in the example above, moov 901 includes geometry tracks 1 to 4 and attribute tracks 1 to 4 corresponding to objects 1 to 4. Unlike file 300, file 900 does not contain a base track, and the information already stored in base track 302 is stored in geometry track 1 (902). In other words, in addition to the information already stored in geometry track 306, geometry track 1 (902) in file 900 also stores common metadata and 3D coordinate transformation information already stored in base track 302 in file 300.

[0074] Typically, using media stored in an ISOBMFF-based format, the playback device can select the track to be played back as needed. Therefore, a file 900 can be generated such that the content required for playing back that file (e.g., background content) is stored in a track (here, geometry track 902) that will necessarily undergo analysis.

[0075] [Second Embodiment] In the foregoing embodiments, Figure 1 The processing units shown are implemented, for example, by dedicated hardware. Some or all of the processing units in the information processing device 100 can be implemented by a computer. In this embodiment, at least a portion of the processing according to the foregoing embodiments is executed by a computer.

[0076] Figure 10 This is a diagram showing the basic configuration of a computer. Figure 10 In this embodiment, the processor 1001, for example, is a CPU, and it controls the overall operation of the computer. The memory 1002, for example, is RAM, and it temporarily stores programs and data. The computer-readable storage medium 1003, for example, is a hard disk or CD-ROM, and it permanently stores programs and data. In this embodiment, the programs storing the functions of each unit stored in the storage medium 1003 are read into the memory 1002. Then, the processor 1001 operates according to the programs in the memory 1002 to implement the functions of each unit.

[0077] exist Figure 10 In this interface, input interface 1004 is used to obtain information from external devices. Output interface 1005 is used to output information to external devices. Bus 1006 connects the various units mentioned above to enable data exchange. Playback device 700 can also be used with... Figure 10 The same hardware shown is used to implement this.

[0078] (Other embodiments) This invention can be implemented by supplying a program for implementing one or more functions of the above embodiments to a system or device via a network or storage medium, and causing one or more processors in the computer of the system or device to read and execute the program. This invention can also be implemented by circuitry (e.g., an ASIC) for implementing one or more functions.

[0079] This invention is not limited to the embodiments described above, and various changes and modifications can be made within the spirit and scope of this invention. Therefore, the following claims are made to inform the scope of this invention.

[0080] This application claims priority to Japanese Patent Application No. 2023-110826, filed on July 5, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device, comprising: A component for obtaining first data of a first three-dimensional object and second data of a second three-dimensional object that is different from the first three-dimensional object; The first generation component is used to generate arrangement information for arranging the first three-dimensional object and the second three-dimensional object in the same coordinate system; The second generation component is used to generate metadata common to the first three-dimensional object and the second three-dimensional object; The third generation component is used to generate a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; as well as The fourth generation component is used to generate a single file storing the first track, the second track, the third track, the first data, the second data, the layout information, and the metadata.

2. The information processing device according to claim 1, characterized in that, The arrangement information includes information indicating the position of each of the three-dimensional objects in the first three-dimensional object and the second three-dimensional object in the same coordinate system, information indicating the pose in the same coordinate system, and information indicating the size in the same coordinate system.

3. The information processing device according to claim 2, characterized in that, The information indicating the position of each three-dimensional object in the first three-dimensional object and the second three-dimensional object in the same coordinate system is a vector between the coordinates of the origin of the local coordinate system of each three-dimensional object in the first three-dimensional object and the coordinates of the origin of the same coordinate system, as evaluated by a three-dimensional orthogonal coordinate system.

4. The information processing device according to claim 2 or 3, characterized in that, Information indicating the pose of each of the three-dimensional objects in the first and second three-dimensional objects in the same coordinate system is the rotation angle between the coordinate axes of the local coordinate system of each of the three-dimensional objects in the first and second three-dimensional objects and the coordinate axes of the same coordinate system.

5. The information processing device according to any one of claims 2 to 4, characterized in that, The information indicating the size of each of the three-dimensional objects in the first and second three-dimensional objects in the same coordinate system is the magnification or reduction factor of the values ​​of the local coordinate system of each of the three-dimensional objects in the first and second three-dimensional objects compared with the values ​​of the X-axis, Y-axis and Z-axis in the same coordinate system, as evaluated by a three-dimensional orthogonal coordinate system.

6. The information processing device according to any one of claims 1 to 5, characterized in that, The origin of the same coordinate system is the reference position of the space managed by the spatial ID.

7. The information processing device according to any one of claims 1 to 6, characterized in that, The metadata is a set of parameters referenced when decoding the first 3D object and the second 3D object.

8. The information processing device according to any one of claims 1 to 7, characterized in that, The third generating component generates boxes that store index information for indicating reference locations of data, serving as each of the first, second, and third tracks. The fourth generation component generates the single file, enabling the same boxes in the box to be used to manage the layout information and the metadata via an index.

9. The information processing device according to claim 8, characterized in that, The second generation component also generates field-of-view information indicating viewpoints in the same coordinate system, and The fourth generation component generates the single file, enabling the metadata and field of view information to be managed via indexing using the same boxes in the boxes.

10. The information processing device according to claim 9, characterized in that, The field of view information is information indicating the position of the viewpoint in the same coordinate system.

11. The information processing apparatus according to any one of claims 8 to 10, characterized in that, The second generating component also generates light source information indicating the light source in the same coordinate system, and The fourth generation component generates the single file, enabling the metadata and light source information to be managed via an index using the same boxes within the boxes.

12. The information processing device according to claim 11, characterized in that, The light source information includes at least one of the following: position, direction, light intensity, and light color of the light source in the same coordinate system.

13. The information processing device according to claim 11 or 12, characterized in that, When the light source includes a spotlight, the light source information includes the light distribution angle of the spotlight.

14. The information processing apparatus according to any one of claims 11 to 13, characterized in that, The light source information refers to information indicating multiple light sources in the same coordinate system.

15. The information processing apparatus according to any one of claims 11 to 14, characterized in that, In the case where the light source includes ambient light that is uniformly incident from all directions, the light source information includes information indicating the proportion of the ambient light relative to the overall light source.

16. A playback device, comprising: A component for obtaining a single file containing data of a first three-dimensional object and data of a second three-dimensional object; A first obtaining component is configured to obtain from the file arrangement information for arranging the first three-dimensional object and the second three-dimensional object in the same coordinate system; The second obtaining component is used to obtain metadata common to the first three-dimensional object and the second three-dimensional object from the file; A decoding component is used to decode the data of the first three-dimensional object and the data of the second three-dimensional object when the first three-dimensional object and the second three-dimensional object can be decoded; as well as A drawing component is used to draw the first three-dimensional object and the second three-dimensional object in space in the same coordinate system based on the arrangement information and the first three-dimensional object and the second three-dimensional object decoded by the decoding component.

17. The playback device according to claim 16, characterized in that, The arrangement information includes information indicating the position of each of the three-dimensional objects in the first three-dimensional object and the second three-dimensional object in the same coordinate system, information indicating the pose in the same coordinate system, and information indicating the size in the same coordinate system.

18. The playback device according to claim 16 or 17, characterized in that, The origin of the same coordinate system is the reference position of the space managed by the spatial ID.

19. The playback device according to any one of claims 16 to 18, characterized in that, The metadata is the level information referenced when decoding the first 3D object and the second 3D object.

20. The playback device according to any one of claims 16 to 19, characterized in that, The second obtaining component also obtains from the document field-of-view information indicating viewpoints in the same coordinate system, and The drawing component draws the first three-dimensional object and the second three-dimensional object based on the field of view information.

21. The playback device according to any one of claims 16 to 20, characterized in that, The second obtaining component also obtains from the document light source information indicating the light source in the same coordinate system, and The drawing component draws the first three-dimensional object and the second three-dimensional object based on the light source information.

22. An information processing method, comprising: Obtain first data of a first three-dimensional object and second data of a second three-dimensional object that is different from the first three-dimensional object; Generate arrangement information for arranging the first 3D object and the second 3D object in the same coordinate system; Generate metadata common to both the first 3D object and the second 3D object; Generate a first track for managing the first data, a second track for managing the second data, and a third track for managing the metadata; as well as Generate a single file that stores the first track, the second track, the third track, the first data, the second data, the layout information, and the metadata.

23. An information processing method, comprising: Obtain a single file containing data of the first three-dimensional object and data of the second three-dimensional object; Obtain arrangement information from the document for arranging the first 3D object and the second 3D object in the same coordinate system; Obtain common metadata for the first 3D object and the second 3D object from the file; If the first 3D object and the second 3D object can be decoded, the data of the first 3D object and the data of the second 3D object are decoded. as well as Based on the layout information and the decoded first and second 3D objects, the first and second 3D objects are drawn in space within the same coordinate system.

24. A program for enabling a computer to function as a component in an information processing apparatus according to any one of claims 1 to 15 or a playback apparatus according to any one of claims 16 to 21.

Images