Scene description editing device and program
The scene description editing device and program facilitate the editing and rendering of content with a degree of freedom of viewpoint by outputting scene descriptions and acquiring specific contents from media processing devices, addressing the challenge of sharing such content between systems.
Patent Information
- Application Number
- JP2023080802
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-28
AI Technical Summary
Existing systems face challenges in effectively sharing and editing scene descriptions that define content with a degree of freedom of viewpoint between media processing devices and user terminals, particularly in handling 360° video and 3D objects.
A scene description editing device and program that outputs scene descriptions defining content with a degree of freedom of viewpoint and acquires specific contents generated by a media processing device based on multiple viewpoints, allowing for appropriate editing and rendering on user terminals.
Enables the appropriate editing of scene descriptions on user terminals, reducing processing load and ensuring seamless rendering of 360° video and 3D objects with a degree of freedom of viewpoint.
Smart Images

Figure 2024165003000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a scene description editing device and a program. [Background technology]
[0002] Conventionally, mechanisms for transmitting content such as 360° video and 3D objects have been proposed (for example, Non-Patent Document 1). Known mechanisms include 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which a user moves their head while in a seated position, and 6DoF, which involves viewpoint movement within the range in which a user can move freely. In such mechanisms, the positional relationship between the 360° video and the 3D object is indicated by a scene description. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] 3GPP TR 26.928 V16.1.0 December 2020 Summary of the Invention [Problem to be solved by the invention]
[0004] In the above-mentioned background, a case may be considered in which specific content including content with viewpoint freedom is generated by a media processing device, and the generated specific content is then transmitted from the media processing device to a user terminal.Similarly, a case may be considered in which a scene description defining the specific content is edited by a scene description editing device.
[0005] As a result of careful consideration, the inventors have found that in the above-mentioned case, it is necessary to standardize the rendering function of the media processing device in the user terminal and the scene description editing device.
[0006] Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide a scene description editing device and program that enables appropriate editing of scene descriptions that define specific content when the rendering function of a media processing device is used in a user terminal. [Means for solving the problem]
[0007] The outline of the disclosure is a scene description editing device comprising an output unit that outputs a scene description and two or more viewpoint information that define the structure of content having viewpoint freedom to a media processing device, and an acquisition unit that acquires two or more specific contents that include at least the content and that have been generated by the media processing device based on the scene description and each of the two or more viewpoint information.
[0008] The outline of the disclosure is a program that causes a computer to execute step A of outputting a scene description and two or more pieces of viewpoint information to a media processing device, the scene description defining the structure of content having viewpoint freedom, and step B of acquiring two or more pieces of specific content that include at least the content, the specific content being generated by the media processing device based on each of the scene description and the two or more pieces of viewpoint information. Effect of the Invention
[0009] According to the present invention, it is possible to provide a scene description editing device and program that enable appropriate editing of a scene description that defines specific content when the rendering function of a media processing device is used in a user terminal. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram showing a transmission system 10 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the invention. [Diagram 3]FIG. 3 is a diagram for explaining the second content according to the embodiment. [Figure 4] FIG. 4 is a diagram showing a method for viewing a specific content according to the embodiment. [Diagram 5] FIG. 5 is a diagram for explaining the first operation example. [Figure 6] FIG. 6 is a diagram for explaining the second operation example. [Figure 7] FIG. 7 is a diagram for explaining the second operation example. [Figure 8] FIG. 8 is a diagram for explaining the third operation example. [Figure 9] FIG. 9 is a diagram for explaining the third operation example. [Figure 10] FIG. 10 is a diagram for explaining the third operation example. [Figure 11] FIG. 11 is a diagram for explaining the third operation example. [Figure 12] FIG. 12 is a diagram for explaining the fourth operation example. [Figure 13] FIG. 13 is a diagram for explaining the fourth operation example. [Figure 14] FIG. 14 is a diagram for explaining the fourth operation example. [Figure 15] FIG. 15 is a diagram for explaining the fifth operation example. [Figure 16] FIG. 16 is a diagram for explaining the fifth operation example. [Figure 17] FIG. 17 is a diagram for explaining the fifth operation example. [Figure 18] FIG. 18 is a diagram for explaining the fifth operation example. [Figure 19] FIG. 19 is a diagram for explaining the first method according to the first modified example. [Figure 20] FIG. 20 is a diagram for explaining the second method according to the first modified example. [Figure 21] FIG. 21 is a block diagram showing a media processing device 200 and a scene description editing device 600 according to the second modification. [Figure 22]FIG. 22 is a diagram showing an example of a UI (User Interface) of a scene description editing device 600 according to the second modified example. [Diagram 23] FIG. 23 is a block diagram showing a media processing device 200 and a scene description editing device 600 according to the third modification. [Figure 24] FIG. 24 is a diagram showing an example of a UI (User Interface) of a scene description editing device 600 according to the third modified example. [Diagram 25] FIG. 25 is a diagram showing an example of a method for outputting two or more specific contents according to the third modification. [Figure 26] FIG. 26 is a diagram showing an example of a method for outputting two or more specific contents according to the third modification. [Figure 27] FIG. 27 is a diagram showing an example of a method for outputting two or more specific contents according to the third modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Next, an embodiment of the present invention will be described. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, it should be noted that the drawings are schematic and the ratios of dimensions may differ from the actual ones.
[0012] Therefore, specific dimensions should be determined with reference to the following description. In addition, the drawings may of course include parts with different dimensional relationships and ratios.
[0013] [Disclosure Summary] The scene description editing device according to the disclosed overview includes an output unit that outputs a scene description that defines a structure of content having viewpoint freedom and two or more viewpoint information to a media processing device, and an acquisition unit that acquires two or more specific contents that include at least the content and that have been generated by the media processing device based on the scene description and each of the two or more viewpoint information.
[0014] In the outline of the disclosure, the scene description editing device acquires two or more specific contents (previews) generated by the media processing device based on the scene description and each of two or more pieces of viewpoint information. With this configuration, assuming a case in which the specific contents are generated (rendered) by the media processing device and the generated specific contents are then transmitted from the media processing device to a user terminal, the scene description editing device can acquire two or more specific contents having different viewpoint information from the media processing device in a similar manner to that of the user terminal. Therefore, for the specific contents expected to be displayed on the user terminal, it is possible to appropriately edit the scene description while checking two or more specific contents (previews) corresponding to the two or more pieces of viewpoint information.
[0015] It should be noted that the specific content generated by the media processing device is generated based on viewpoint information, and that the video included in the specific content can be treated as 2D video on the user terminal side.
[0016] [Embodiment] (Transmission System) A transmission system according to an embodiment will be described below. Fig. 1 is a diagram showing a transmission system 10 according to an embodiment. As shown in Fig. 1, the digital wireless transmission system includes a transmitting device 100, a media processing device 200, and a user terminal 300.
[0017] In an embodiment, the transmitting device 100 transmits a first content having no viewpoint freedom and a second content having viewpoint freedom to the media processing device 200. Furthermore, the transmitting device 100 transmits to the media processing device 200 first control information associated with the first content and second control information associated with the second content.
[0018] The first content may include at least one of 2D video and audio. The first content and the first control information may be transmitted in a first method. The first method may be a method conforming to ISO / IEC 23008-1 (hereinafter, MMT (MPEG Media Transport)). In the following, a case will be illustrated in which the first method is MMTP (MMT Protocol) conforming to MMT. The first control information may be referred to as MMT-SI (Signaling Information).
[0019] The second content may include a 360° video and a 3D object. The second content and the second control information may be transmitted in a second method. Alternatively, they may be transmitted in a protocol such as HTTP (Hyper Text Transfer Protocol). The second content may conform to 3DoF+ (Degree of Freedom) with viewpoint movement within the range in which a user moves his / her head while sitting, or 6DoF with viewpoint movement within the range in which a user can move freely. Since the second content has viewpoint freedom, it may include two or more 360° videos and two or more 3D objects at the same time (frame). The second control information may be referred to as a scene description.
[0020] Here, the second control information may be transmitted in the above-mentioned first method. That is, the second control information may be transmitted in the same first method (e.g., MMTP) as the first control information. Alternatively, the second control information may be transmitted in a protocol such as HTTP.
[0021] The transmission from the transmitting device 100 to the media processing device 200 is not particularly limited, but may be transmission using satellite broadcasting, transmission using the Internet network, or transmission using a mobile communications network.
[0022] Although not particularly limited, the transmission system may be a digital wireless transmission system. The digital wireless transmission system may be a system used for 4K and 8K satellite broadcasting.
[0023] Media processing device 200 generates specific content including at least the above-mentioned second content based on the viewpoint information received from user terminal 300, and transmits the generated specific content to user terminal 300. Although not particularly limited, transmission of the specific content may be transmission using the Internet network or a mobile communication network.
[0024] The user terminal 300 may be a user terminal such as a smartphone, a tablet terminal, or a head-mounted display. As shown in Fig. 1, two or more user terminals 300 may be provided as the user terminal 300. In other words, two or more user terminals 300 may request the media processing device 200 to generate specific content. Each user terminal 300 may transmit different viewpoint information to the media processing device 200.
[0025] (Media processing device and user terminal) A media processing device and a user terminal according to an embodiment of the present invention will now be described. Fig. 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the present invention.
[0026] First, the media processing device 200 includes a receiving unit 210, a renderer 220, and an encoding processing unit 230.
[0027] The reception unit 210 receives viewpoint information. In the embodiment, the reception unit 210 constitutes a receiving unit that receives the viewpoint information from the user terminal 300. The viewpoint information includes an information element indicating the viewpoint position of the user of the user terminal 300 and an information element indicating the line of sight of the user of the user terminal 300.
[0028] The renderer 220 generates specific content including at least the second content based on the viewpoint information. Since the specific content is generated based on the viewpoint information, it may include one 360° video and one 3D object at the same time (frame). In the following, a case where the specific content includes the first content in addition to the second content will be illustrated.
[0029] 2, the renderer 220 generates a first content including 2D video and audio as a part of a specific content based on the first control information (MMT-SI). Viewpoint information is not required in generating the first content.
[0030] Specifically, the renderer 220 acquires the 2D video, audio, and MMT-SI in the form of MMTP packets in which the 2D video, audio, and MMT-SI are packetized.
[0031] For example, the MMTP packet is stored in an IP (Internet Protocol) packet, which may be transmitted using UDP (User Datagram Protocol) or TCP (Transmission Control Protocol).
[0032] Here, the first content is processed in units (hereinafter, MPU; Media Processing Unit) that are separated by a certain time width. The MPU includes one or more access units. The access unit may be treated as an MFU (Media Fragment Unit). The MFU related to 2D video may be called a NAL (Network Abstraction Layer) unit, and the MFU related to audio may be called an MHAS (MPEG-H 3D Audio Stream) packet.
[0033] The MMT-SI includes a PA (Package Access) message, and the PA message includes an MPT (MMT Package Table) indicating a list of the first content. Furthermore, the MMT-SI includes an MPU timestamp descriptor indicating the presentation time of the first content. The MPU timestamp descriptor may mean the presentation time of the MPU, that is, the time of the access unit that is first presented in the MPU.
[0034] The MPU timestamp descriptor may be generated using UTC (Coordinated Universal Time) as the reference time. The reference time may be TAI (International Atomic Time) or may be a time provided by a GPS (Global Positioning System). The reference time may be a time provided by an NTP (Network Time Protocol) server or a PTP (Precision Time Protocol) server.
[0035] 2, the renderer 220 generates a second content including a 360° video image and a 3D object as a part of the specific content based on the second control information (scene description). Viewpoint information is used in generating the second content.
[0036] Specifically, the renderer 220 may acquire the scene description in the form of an MMTP packet in which the scene description is packetized. There is no particular limitation on the method of acquiring the 360° video and the 3D object.
[0037] The 360° video may be converted into a 2D video by a projective transformation such as ERP (Equirectangular projection) or cube map. Metadata indicating the type of projective transformation applied to the 360° video may be added. The 3D object may be encoded in a mesh format. For the mesh format encoding, ISO / IEC 14496-16 "Animation framework extension (AFX)" may be used. The 3D object may be encoded in a point cloud format. For the point cloud format encoding, ISO / IEC 23090-5 "Video-based Point Cloud Compression" may be used.
[0038] Here, the second content is collected into one file in units separated by a certain time interval. The certain time interval may be 500 ms. For example, when the frame rate is 60 fps (frames per second), one file includes 30 frames.
[0039] The scene description is generated for each file and includes information for identifying the 360° video and the 3D object for each frame. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, and an information element (scale_object) indicating the size of the 3D object in the frame.
[0040] Third, the renderer 220 outputs the specific content including the first content and the second content to the encoding processing unit 230. The renderer 220 may output the presentation time of the specific content to the encoding processing unit 230 together with the specific content.
[0041] Here, the presentation time of the specific content may be corrected based on the delay time between the media processing device 200 and the user terminal 300. Specifically, the renderer 220 may calculate the presentation time (T'=T+ΔT) of the specific content to be provided from the media processing device 200 to the user terminal 300 based on the presentation time (T) and delay time (ΔT) of the specific content provided from the transmitting device 100 to the media processing device 200. The delay time (ΔT) may be a value that is determined in advance in the media processing device 200, or may be a different value for each user terminal 300.
[0042] Fourthly, the renderer 220 may constitute a transmission unit that transmits viewpoint information used in generating the specific content to the user terminal 300. The viewpoint information used in generating the specific content may be transmitted from the encoding processing unit 230 to the user terminal 300.
[0043] For example, the transmission method of the viewpoint information and the specific content may be MMTP or HTTP. When MMTP is used as the transmission method of the specific content, the viewpoint information may be stored as metadata in OMAF (Omnidirectional Media Format) defined in ISO / IEC 23090-2.
[0044] The encoding processing unit 230 encodes the specific content generated by the renderer 220. In the embodiment, the encoding processing unit 230 may be an example of a transmission unit that transmits the specific content to the user terminal 300.
[0045] Furthermore, the encoding unit 230 may encode the presentation time of the specific content. The encoding unit 230 may transmit an information element indicating the presentation time to the user terminal 300 together with the specific content.
[0046] Here, any compression encoding method can be used as the compression encoding method used by the encoding processing unit 230. For example, the compression encoding method may be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).
[0047] As described above, the second content included in the specific content is generated based on viewpoint information, and therefore the video included in the specific content can be treated as a 2D video that does not have freedom of viewpoint.
[0048] For example, the transmission control method used to start and end viewing of the specific content may include RTSP (Real Time Streaming Protocol). The transmission method may be MMTP or HTTP. When MMTP is used as the transmission method, the specific content may be stored in OMAF defined in ISO / IEC 23090-2.
[0049] As shown in FIG. 2, the user terminal 300 includes a detection unit 310, a decoding processing unit 320, and a renderer 330.
[0050] The detection unit 310 detects the user's viewpoint position and line of sight direction. The detection unit 310 may include an acceleration sensor or a Global Positioning System (GPS) sensor. The detection unit 310 may include a user I / F (e.g., a touch sensor, a keyboard, a mouse, a controller, etc.) that is manually input by the user. The detection unit 310 may transmit viewpoint information (viewpoint position and line of sight direction) to the media processing device 200. The detection unit 310 may output the viewpoint information (viewport) to the renderer 330.
[0051] The decoding processing unit 320 decodes the specific content received from the media processing device 200. The decoding processing unit 320 may decode the presentation time received from the media processing device 200. The decoding processing unit 320 may output the specific content to the renderer 330, and may also output the presentation time to the renderer 330.
[0052] The renderer 330 outputs the specific content decoded by the decoding processing unit 320. The renderer 330 may output the specific content based on the presentation time decoded by the decoding processing unit 320. For example, the renderer 330 may output the video content included in the specific content to a display and output the audio content included in the specific content to a speaker.
[0053] Here, the renderer 330 may generate specific content in which the viewpoint position and line of sight direction have been modified based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310 .
[0054] (Second content) The second content according to the embodiment will be described below. Here, the second content at t=0, t=1, and t=2 will be described. The time intervals between t=0, t=1, and t=2 are not particularly limited.
[0055] For example, as shown in Fig. 3, at t=0, a 360° video may be displayed without a 3D object being displayed. The 360° video may be considered to be a background video of the 3D object. At t=1, a 3D object may be displayed in a form superimposed on the 360° video. Furthermore, at t=1, the position and rotation of the 3D object superimposed on the 360° video may be changed.
[0056] The above-mentioned scene description includes information elements indicating the position, rotation, and size of a 3D object for each of t=0, t=1, and t=2, and allows the 3D object to be appropriately superimposed on a 360° video image.
[0057] (How to watch) A viewing method according to the embodiment will be described below, in which viewing of a specific content including a first content and a second content will be exemplified.
[0058] 4, in step S11, the user terminal 300 transmits an RTSP SETUP to the media processing device. The RTSP SETUP is a message indicating that viewing of specific content is to begin.
[0059] Here, the RTSP SETUP includes the IP address of the user terminal 300, the listening port number, and content identification information (content ID). The RTSP SETUP may include capability information of the user terminal 300 for viewing specific content. The capability information may include the frame rate, the display resolution, and the like. The display resolution may include the field of view (FoV). The capability information may include information elements indicating the encoding method and the compression method supported by the user terminal 300.
[0060] Here, a case is illustrated in which the capability information of the user terminal 300 is directly notified to the media processing device 200, but the embodiment is not limited to this. The capability information of the user terminal 300 may be notified to the transmitting device 100, and then the transmitting device 100 may notify the media processing device 200.
[0061] In step S12, the media processing device 200 transmits a response to the RTSP SETUP. Here, an ACK indicating that the RTSP SETUP has been accepted is transmitted as the response.
[0062] In step S21, the user terminal 300 transmits initial viewpoint information to the media processing device 200. The initial viewpoint information may be transmitted in the MMT-SI format.
[0063] In step S22, the media processing device 200 generates initial specified content based on the initial viewpoint information (rendering process). For example, the media processing device 200 generates second content to be included in the initial specified content based on the initial viewpoint information and the scene description.
[0064] Here, media processing device 200 may generate initial specific content using as a viewport a range wider than the display resolution of user terminal 300. For example, the range wider than the display resolution may be a range that is the display resolution + 20% in the horizontal direction and the display resolution + 20% in the vertical direction.
[0065] The media processing device 200 applies a compression encoding method to the initial specific content. Although not particularly limited, the compression encoding method may be HEVC or VVC.
[0066] In step S23, the media processing device 200 transmits the initial specified content corresponding to the initial viewpoint information to the user terminal 300. The media processing device 200 transmits the presentation time of the initial specified content to the user terminal 300. As described above, the presentation time (T') provided to the user terminal 300 may be determined based on the delay time (ΔT).
[0067] If a different value is used for the delay time (ΔT) for each user terminal 300, it is possible for the media processing device 200 to specify this by including the transmission time of the RTSP SETUP in the above-mentioned RTSP SETUP.
[0068] The user terminal 300 outputs the specific content based on the presentation time (T'). The user terminal 300 may generate specific content with a corrected viewpoint position and line of sight direction based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310.
[0069] In step S31, the user terminal 300 transmits viewpoint information to the media processing device 200. The viewpoint information may be transmitted in the MMT-SI format. Here, the user terminal 300 may transmit the viewpoint information at a predetermined cycle (for example, 500 ms), or may transmit the viewpoint information in response to a change in at least one of the viewpoint position and the line of sight direction.
[0070] In step S32, media processing device 200 generates specific content based on the viewpoint information received in step S31 (rendering process).
[0071] In step S33, media processing device 200 transmits to user terminal 300 specific content that corresponds to the viewpoint information received in step S31.
[0072] The processing of steps S31 to S33 is the same as that of steps S21 to S23, except that the viewpoint information received in step S31 is used instead of the initial viewpoint information. Therefore, details of the processing of steps S31 to S33 will be omitted. The processing of steps S31 to S33 may be repeated at a predetermined cycle, or may be repeated every time the user's viewpoint position or line of sight changes.
[0073] In step S41, the user terminal 300 transmits an RTSP TEARDOWN to the media processing device. The RTSP TEARDOWN is a message indicating that viewing of a particular content is to be terminated.
[0074] In step S42, the media processing device 200 transmits a response to the RTSP TEARDOWN. Here, an ACK indicating that the RTSP TEARDOWN has been accepted is transmitted as the response.
[0075] 4 illustrates an example in which steps S11 and S12 are performed on an RTSP basis, but the embodiment is not limited to this. Steps S11 and S12 may be performed on an MMTP basis or on an HTTP basis.
[0076] Similarly, although the case where step S41 and step S42 are executed based on RTSP has been exemplified, the embodiment is not limited thereto. Step S41 and step S42 may be executed based on MMTP or HTTP.
[0077] 4, the case where steps S31 to S33 are executed based on the MMTP is illustrated, but the embodiment is not limited to this. Steps S31 to S33 may be executed based on another method (for example, HTTP).
[0078] Similarly, although the case where steps S41 to S43 are executed based on the MMTP has been exemplified, the embodiment is not limited to this, and steps S41 to S43 may be executed based on another method (for example, HTTP).
[0079] (Example 1) The above-described embodiment may include the following Operation Example 1. In Operation Example 1, the media processing device 200 transmits to the user terminal 300 viewpoint information used in generating the specific content, in association with a sequence number added to the specific content.
[0080] Specifically, the media processing device 200 (renderer 220) generates second content including a 360° video image and a 3D object as part of a specific content based on the second control information (scene description) as in the above-described embodiment. In generating the second content, viewpoint information received from the user terminal 300 is used.
[0081] In the operation example 1, the renderer 220 associates viewpoint information used in generating a specific content (here, the second content) with a sequence number. The renderer 220 transmits the viewpoint information used in generating the specific content to the user terminal 300 in association with a sequence number added to the specific content. The viewpoint information may be stored in a VP (View Port) message. The VP message may have a format of MMT-SI defined in ISO / IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc. The VP message may be transmitted for each frame. The VP message may be transmitted to the user terminal 300 as a message (MMT-SI) related to MMTP.
[0082] Although not particularly limited, the VP message may have a data structure shown in Fig. 5. As shown in Fig. 5, the VP message may include message_id, version, length, fov, viewpoint_pos_x, viewpoint_pos_y, viewpoint_pos_z, viewpoint_yaw, viewpoint_pitch, viewpoint_roll, viewport_width, viewport_height, mpu_sequence_number_flag, mpu_sequence_number, etc.
[0083] The message_id is identification information indicating a VP message. The message_id may be 0x0204.
[0084] The version is information indicating the version of the MMTP protocol. The version may be 0x00.
[0085] The length is information indicating the length of the VP message.
[0086] fov is information indicating the field of view.
[0087] viewpoint_pos_x is information indicating the x-coordinate of the viewpoint position. viewpoint_pos_x is an example of viewpoint information used in generating a specific content.
[0088] viewpoint_pos_y is information indicating the y coordinate of the viewpoint position. viewpoint_pos_y is an example of viewpoint information used in generating specific content.
[0089] viewpoint_pos_z is information indicating the z coordinate of the viewpoint position. Viewpoint_pos_z is an example of viewpoint information used in generating a specific content.
[0090] viewpoint_yaw is information indicating the yaw of the viewpoint position. Viewpoint_yaw is an example of viewpoint information used in generating specific content.
[0091] viewpoint_pitch is information indicating the pitch of the viewpoint position. Viewpoint_pitch is an example of viewpoint information used in generating a specific content.
[0092] The viewpoint_roll is information indicating the role of the viewpoint position. The viewpoint_roll is an example of viewpoint information used in generating a specific content.
[0093] The viewport_width is information indicating the width of the display area (specific content).
[0094] The viewport_height is information indicating the height of the display area (specific content).
[0095] The mpu_sequence_number_flag is information indicating whether or not the mpu_sequence_number field exists. For example, when the mpu_sequence_number_flag is 1, the mpu_sequence_number field exists, and when the mpu_sequence_number_flag is 0, the mpu_sequence_number field does not necessarily exist.
[0096] The mpu_sequence_number is an MPU sequence number of a video corresponding to a specific content indicated by the VP message, and is an example of a sequence number added to a specific content.
[0097] Here, viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z are examples of information elements indicating the user's viewpoint position in the three-dimensional space configured by the scene description. viewpoint_yaw, viewpoint_pitch, and viewpoint_roll are examples of information elements indicating the user's line of sight direction in the three-dimensional space configured by the scene description. viewport_width and viewport_height are examples of information elements indicating the number of pixels of an image included in a specific content.
[0098] Under these conditions, the media processing device 200 and user terminal 300 may execute the following operations.
[0099] First, the media processing device 200 (renderer 220) may determine viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z based on the viewpoint position used to generate the specific content. The renderer 220 may determine viewpoint_yaw, viewpoint_pitch, and viewpoint_roll based on the line of sight direction used to generate the specific content. The renderer 220 may determine viewport_width based on the number of pixels in the horizontal direction of the specific content, and may determine viewport_height based on the number of pixels in the vertical direction of the specific content.
[0100] The media processing device 200 (encoding processing unit 230) may execute compression encoding of the specific content generated by the renderer 220, and transmit the specific content. Here, the encoding processing unit 230 may transmit the VP message shown in Fig. 5 to the user terminal 300. In other words, the encoding processing unit 230 may transmit the viewpoint information used in generating the specific content to the user terminal 300, in association with the sequence number added to the specific content.
[0101] Secondly, the user terminal 300 (the decoding processing unit 320) may constitute a receiving unit that receives viewpoint information used in generating a particular content from the media processing device 200 in association with a sequence number added to the particular content. In other words, the decoding processing unit 320 may receive a VP message (MMT-SI) shown in FIG. 5 from the media processing device 200.
[0102] The user terminal 300 (renderer 330) may associate the video constituting the decoded specific content with the viewpoint information included in the VP message based on the sequence number (mpu_sequence_number). The renderer 330 may specify a video in a range specified by the viewpoint information detected by the detection unit 310 from a display area defined by information included in the VP message (viewport_width and viewport_height) based on a difference between the viewpoint information detected by the detection unit 310 and the viewpoint information included in the VP message. The renderer 330 may display the specified video.
[0103] (Example 2) The above-described embodiment may include the following operation example 2. In the operation example 2, as shown in FIG. 6, a case is assumed in which a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed. The first user terminal 400 may be a terminal such as a head-mounted display. The first user terminal may have the same function as the above-described user terminal 300. The second user terminal 500 may be a terminal such as a volumetric display.
[0104] Specifically, in the second operational example, as shown in FIG. 6, the media processing device 200 (renderer 220) may constitute a receiving unit that receives a content configuration and recommended viewport information from the transmitting device 100. The content configuration may be considered to include 2D video, audio, 360° video, and 3D objects. The content configuration may be considered to include MMT-SI and a scene description. The recommended viewport information may be considered to be an example of specific viewpoint information. The recommended viewport information may be information that defines (recommends) the position, direction, and angle of view at which a video is viewed in a three-dimensional space (a three-dimensional space constructed by a scene description) constituted by a specific content. The recommended viewport information may include an information element indicating at least one of an information element indicating a viewpoint position in a three-dimensional space constituted by a specific content and an information element indicating a line-of-sight direction in a three-dimensional space constituted by a specific content. The recommended viewport information may be considered to be viewpoint information used primarily in the second user terminal 500.
[0105] In the following, unless explicitly stated, the viewpoint information used in generating specific content may include specific viewpoint information (recommended viewport information) received from the transmitting device 100, or may include viewpoint information received from the user terminal 300.
[0106] Although not particularly limited, the recommended viewport information may be included in the scene description in a manner shown in Fig. 7. As shown in Fig. 7, the recommended viewport information may include camera_orientation, frame_number, translation, and yfov.
[0107] The camera_orientation is information that indicates the direction in which the image is viewed in the three-dimensional space constructed by the scene description. The camera_orientation may be considered to be synonymous with the line of sight direction.
[0108] The frame_number is information indicating the frame number of the video to which the camera_orientation, translation, and yfov are applied. The camera_orientation may be considered to be an example of specific viewpoint information.
[0109] Translation is information indicating the position at which an image is viewed in a three-dimensional space constructed by a scene description. Translation may be considered to be synonymous with viewpoint position. Translation may be considered to be an example of specific viewpoint information. For example, FIG. 7 illustrates a case in which the viewpoint position is [0,0,-50] when the frame number is 0, and the viewpoint position moves to [0,0,-75] when the frame number is 2505.
[0110] yfov is information that indicates the angle of view from which an image is viewed in a three-dimensional space constructed by a scene description.
[0111] Although not limited thereto, the camera_orientation and translation may be provided by the creator of the content, or, assuming a case where the content is captured by a camera, the camera_orientation and translation may be automatically provided by a GPS and a sensor provided in the camera.
[0112] Here, translation is an example of an information element that indicates the viewpoint position in a three-dimensional space formed by a specific content, and camera_orientation is an example of an information element that indicates the line of sight direction in a three-dimensional space formed by a specific content.
[0113] Under these conditions, the media processing device 200 and user terminal 300 may execute the following operations.
[0114] First, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information). The media processing device 200 (encoding processing unit 230) may execute compression encoding of the specific content generated by the renderer 220 and transmit the specific content. Here, the encoding processing unit 230 may transmit the specific content generated based on the specific viewpoint information to the first user terminal 400, or may transmit the specific content generated based on the specific viewpoint information to the second user terminal 500.
[0115] Secondly, the media processing device 200 (receiving unit 210) may receive viewpoint information from the first user terminal 400 when the user terminal is the first user terminal 400 that feeds back viewpoint information. The media processing device (renderer 220) may generate specific content based on the viewpoint information received from the first user terminal 400. In such a case, the media processing device 200 (receiving unit 210) may receive a reset signal from the first user terminal 400. The media processing device (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information) in response to the reset signal.
[0116] In such a case, the first user terminal 400 may have a similar configuration to the user terminal 300. However, the first user terminal 400 (detection unit 310) may have a function for detecting a reset signal. The detection unit 310 may detect a user operation that inputs a reset signal. The detection unit 310 may transmit the reset signal to the media processing device 200.
[0117] Thirdly, when the user terminal is a second user terminal 500 that does not feed back viewpoint information, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information).
[0118] In such a case, the second user terminal 500 may not have a detection unit 310 that detects viewpoint information. The second user terminal 500 may not have a renderer 330. The second user terminal 500 may have the same configuration as the user terminal 300, except that the second user terminal 500 does not have the detection unit 310 and the renderer 330.
[0119] (Example 3) The above-described embodiment may include the following Operation Example 3. In Operation Example 3, media processing device 200 (for example, selection unit 260 described below) may constitute a receiving unit that receives, from transmitting device 100, quality information on each of two or more streams whose quality differs depending on the orientation of the 3D object based on viewpoint information, as quality information on a stream related to a 3D object included in specific content.
[0120] Specifically, in operation example 3, as shown in Fig. 8, the media processing device 200 has a selection unit 260 in addition to the components shown in Fig. 2. The selection unit 260 receives a scene description and a 3D object from the transmitting device 100. The selection unit 260 inputs a stream (3D object) selected from two or more streams to the renderer 220. The selection unit 260 may request the transmitting device 100 to transmit the selected stream. Note that in a case where the media processing device 200 transmits specific content to multiple user terminals 300, the selection unit 260 may request the transmitting device 100 to transmit a stream required by each of the multiple user terminals 300, or may request the transmitting device 100 to transmit all of the streams.
[0121] Here, the selection unit 260 receives, from the transmission device 100, quality information on each of two or more streams whose quality differs depending on the orientation of the 3D object based on the viewpoint information, as quality information on the stream related to the 3D object.
[0122] The quality information may be information indicating the relative quality of each face constituting a bounding box for a 3D object. The bounding box may be represented by a three-dimensional rectangle onto which the 3D object is projected. For example, as shown in FIG. 9, the bounding box may be defined by vertices A to H. In such a case, the faces of the bounding box include face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, face #3 represented by vertices A, B, C, and D, face #4 represented by vertices E, F, G, and H, face #5 represented by vertices A, D, H, and E, and face #6 represented by vertices D, C, G, and H.
[0123] In such a case, when we consider the case of viewing a 3D object in the 3D space constructed by the scene description, we assume that three sides are primarily observed, or in other words, that the remaining three sides are not observed very often.
[0124] In the third operational example, two or more streams with different qualities depending on the orientation of the 3D object based on the viewpoint information are prepared as streams related to a 3D object included in a particular content.
[0125] Although not particularly limited, the quality information may be included in the scene description in the form shown in FIG. 10. In FIG. 10, six streams are illustrated as streams with different orientations of 3D objects. The quality information may be expressed in the form of "quality" [#1, #2, #3, #4, #5, #6]. In the [ ], #1 to #6 mean the quality indexes of surfaces #1 to #6. The quality index may take a value in the range of 1 to 9. A higher value of the quality index may mean a higher quality. For example, in a stream identified by "id"="1", the quality ("8") of #1, #2, #3 is high, and the quality ("3") of #4, #5, #6 is low. In a stream identified by "id"="2", the quality ("3") of #1, #2, #3 is low, and the quality ("8") of #4, #5, #6 is high.
[0126] Under these conditions, the media processing device 200 may execute the following operations: In the following, the selection of a stream (3D object) selected from two or more streams will be mainly described.
[0127] In mode 1, as shown in the upper part of Fig. 11, the media processing device 200 (selector 260) may identify the vertex closest to the user's viewpoint (for example, vertex B), and then identify the three faces that have the closest vertex (for example, face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). The selector 260 may select the stream that maximizes the sum of the quality indexes of the three identified faces (the stream identified by "id"="1" in the example shown in Fig. 10).
[0128] In addition, in mode 1, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information and quality information.
[0129] Mode 2 may be a mode applied in cases where a reduced or enlarged display of a 3D object is performed. For example, as shown in the middle part of FIG. 11, the media processing device 200 (selection unit 260) may identify the vertex (e.g., vertex B) closest to the user's viewpoint position, and then identify three faces having the closest vertex (e.g., face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). In cases where a reduced display of a 3D object is performed, pixels of the 3D object are thinned out. Therefore, the selection unit 260 may select a stream that minimizes the sum of the quality indexes of the three identified faces (the stream identified by "id"="2" in the example shown in FIG. 10). On the other hand, in cases where an enlarged display of a 3D object is performed, pixels of the 3D object are interpolated. Therefore, the selection unit 260 may select a stream that maximizes the sum of the quality indexes of the three identified faces (the stream identified by "id"="1" in the example shown in FIG. 10).
[0130] In addition, in mode 2, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information and quality information.
[0131] Mode 3 may be a mode applied in a case where two 3D objects (3D object #1 and 3D object #2) overlap in the user's line of sight. Here, the selection of a stream related to 3D object #1 will be described. For example, as shown in the lower part of FIG. 11, the media processing device 200 (selection unit 260) may identify the vertex (e.g., vertex B) closest to the user's viewpoint position, and then identify three faces having the closest vertex (e.g., face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). Here, 3D object #2 overlaps on a line segment connecting the vertex (e.g., vertex B) closest to the user's viewpoint position and the user's viewpoint position, and the identified three faces are blocked by 3D object #2. Therefore, the selection unit 260 may select a stream (in the example shown in FIG. 10, the stream identified by "id"="2") for which the sum of the quality indexes of the identified three faces is the smallest.
[0132] In addition, in mode 3, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, it may be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information and the quality information. Furthermore, in mode 3, since the overlap of two 3D objects is identified based on the viewpoint information and the arrangement information of the 3D objects, it may be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information, the quality information, and the arrangement information. The arrangement information of the 3D objects (for example, "rotation_object", "scale_object", and "translation_object" shown in FIG. 10) may be included in the scene description. The arrangement information of the 3D objects may be considered to be "link_area" shown in FIG. 10. That is, the selection unit 260 may receive the arrangement information of the 3D objects in the three-dimensional space from the transmission device 100.
[0133] In the quality information shown in Fig. 10, the sum of the quality indexes of the six aspects is equal in each stream. However, the embodiment is not limited to this. The sum of the quality indexes of the six aspects may be different between two or more streams.
[0134] (Example 4) The above-described embodiment may include the following Operation Example 4. Here, Operation Example 4 includes the operations described below in addition to Operation Example 3. In Operation Example 4, media processing device 200 (e.g., selection unit 270 described below) may constitute a receiving unit that receives importance information for each of two or more objects included in specific content from transmitting device 100.
[0135] Specifically, in operation example 4, as shown in Fig. 12, the media processing device 200 has a selection unit 270 in addition to the components shown in Fig. 2. The selection unit 270 receives a scene description, a 3D object, and a 360° video from the transmission device 100. The selection unit 270 inputs a stream (3D object) selected from two or more streams to the renderer 220.
[0136] Here, the selection unit 270 receives importance information regarding each of the two or more objects from the transmission device 100. The objects may include a 3D object and a 360-degree video.
[0137] For example, the importance information may be information indicating the relative importance between two or more objects. For example, as shown in FIG. 13, consider a case in which object A (background), object B (person), and object C (dog) exist in a three-dimensional space constructed by a scene description. Object A (background) is an example of a 360° image, and object B (person) and object C (dog) are examples of 3D objects. In such a case, the importance information may be information indicating the relative importance between each of object A (background), object B (person), and object C (dog).
[0138] Although not particularly limited, the importance information may be included in the scene description in a form shown in FIG. 14. In FIG. 14, the importance information may be represented by weight. Weight may take a value in the range of 1 to 9. A larger weight value may indicate a higher importance. FIG. 14 illustrates a case in which the weight ("9") of object A (background) identified by "object_id"="0" is the highest, the weight ("3") of object B (person) identified by "object_id"="1" is the lowest, and the weight ("8") of object C (dog) identified by "object_id"="2" is higher than the weight of object B (person) and lower than the weight of object A (background).
[0139] Under these conditions, the media processing device 200 may execute the following operations: In the following, the selection of a stream (3D object) selected from two or more streams will be mainly described.
[0140] First, the media processing device 200 (selector 270) selects the highest quality stream for the 3D object with the highest importance. The stream selection method may be the same as in operation example 3. For example, because the importance of object C (dog) is greater than the importance of object B (person), the selector 270 selects, for object C (dog), the stream with the largest sum of the quality indexes of the three faces that have the vertex closest to the user's viewpoint.
[0141] Secondly, the media processing device 200 (selection unit 270) selects the lowest quality stream for 3D objects other than the 3D object with the highest importance. Next, the selection unit 270 replaces the lowest quality stream with a higher quality stream within the scope in which the specific condition is satisfied, in order from the 3D object with the highest importance. The specific condition may include a first condition that the bandwidth of the line from the transmitting device 100 to the media processing device 200 is equal to or lower than a threshold, and may include a second condition that the processing load of the media processing device 200 is equal to or lower than a threshold. The specific condition may be defined by a combination of the first and second conditions. For example, because the importance of object B (person) is lower than the importance of object C (dog), the selection unit 270 selects a high quality stream for object B (person) within the scope in which the specific condition is satisfied.
[0142] As described above, it may be considered that media processing device 200 (selector 270) selects from two or more streams the stream to be transmitted to user terminal 300 based on viewpoint information, quality information, and importance information. It may be considered that media processing device 200 (selector 270) selects from two or more streams the stream to be transmitted to user terminal 300 based on viewpoint information, quality information, placement information, and importance information.
[0143] In the fourth operational example, a case where one stream exists for the 360° video has been exemplified. However, the embodiment is not limited to this. For the 360° video, two or more streams of different qualities may exist.
[0144] In the operation example 4, a case where two or more streams having different qualities exist for a 3D object depending on the orientation of the 3D object based on viewpoint information is illustrated. However, the embodiment is not limited to this. Two or more streams having different qualities may exist for a 3D object regardless of the orientation of the 3D object.
[0145] (Example 5) The above-described embodiment may include the following Operation Example 3. In Operation Example 3, the media processing device 200 (e.g., the renderer 220) may constitute a receiving unit that receives, from the transmitting device 100, an information element that defines a movement range of a user's viewpoint position in a three-dimensional space configured by specific content (a three-dimensional space constructed by a scene description).
[0146] First, in the fifth operational example, the information element may include an information element (hereinafter, a first information element) that restricts the movement of the user's viewpoint position to the inside of a 3D object included in a specific content. For example, as shown in Fig. 15, in a case where a 3D object is placed in a three-dimensional space constructed by a scene description, the movement of the viewpoint position to the inside of the 3D object may be restricted. However, in a case where the 3D object is a building or another scene exists inside the 3D object, the movement of the viewpoint position to the inside of the 3D object may be permitted.
[0147] Secondly, in the fifth operational example, the information element may include an information element (hereinafter, second information element) that restricts the movement of the user's viewpoint position outside the three-dimensional space. For example, as shown in FIG. 16, the three-dimensional space may be defined by a combination of a rectangular parallelepiped and a spheroid. The number of rectangular parallelepipeds that define the three-dimensional space may be two or more, and the number of spheroids that define the three-dimensional space may be two or more. However, there may be cases in which the movement of the user's viewpoint position outside the three-dimensional space is permitted.
[0148] Although not particularly limited, the first information element may be included in the scene description in a manner shown in Fig. 17. In Fig. 17, the first information element may be represented by viewing_inside_object_flag. The viewing_inside_object_flag may be set for each 3D object. For example, when the viewing_inside_object_flag is "0", movement of the viewpoint position into the 3D object may be restricted, and when the viewing_inside_object_flag is "1", movement of the viewpoint position into the 3D object may be permitted.
[0149] Although not particularly limited, the second information element may be included in the scene description in a manner shown in Fig. 18. In Fig. 18, the second information element may include information elements (cuboid_center_x, cuboid_center_y, cuboid_center_z, cuboid_size_x, cuboid_size_y, cuboid_size_z) that define a rectangular parallelepiped that defines a three-dimensional space. The cuboid_center_x, cuboid_center_y, and cuboid_center_z are information elements that indicate the center position of the rectangular parallelepiped, and the cuboid_size_x, cuboid_size_y, and cuboid_size_z are information elements that indicate the size of the rectangular parallelepiped. The second information element may include information elements (spheroid_center_x, spheroid_center_y, spheroid_center_z, spheroid_size_x, spheroid_size_y, spheroid_size_z) that define a spheroid that defines the three-dimensional space. The spheroid_center_x, spheroid_center_y, and spheroid_center_z are information elements indicating the center position of a spheroid, and the spheroid_size_x, spheroid_size_y, and spheroid_size_z are information elements indicating the size of the spheroid. Note that the cuboid_enable is an information element indicating whether or not a three-dimensional space is defined by a rectangular parallelepiped, and the spheroid_enable may be an information element indicating whether or not a three-dimensional space is defined by a spheroid. FIG. 18 illustrates a case in which a three-dimensional space is defined by two rectangular parallelepipeds and two spheroids.
[0150] Under these conditions, the media processing device 200 (renderer 220) may execute the following operations.
[0151] First, when the user's viewpoint position moves outside the movement range, the renderer 220 may generate specific content by using the intersection point of the path of the user's viewpoint position and the boundary of the movement range as the viewpoint position. In other words, the renderer 220 may fix the viewpoint position at the position (boundary position) at the time when the viewpoint position is about to move outside the movement range.
[0152] Second, when the user's viewpoint position moves outside the movement range, the renderer 220 may notify the user that the movement of the viewpoint position is restricted. For example, the renderer 220 may display a message such as "You cannot move beyond this point."
[0153] (Action and Effects) In the embodiment, media processing device 200 generates specific content based on viewpoint information and transmits the specific content to user terminal 300. With this configuration, there is no need for user terminal 300 to generate specific content including second content with viewpoint freedom, and user terminal 300 can present the specific content by simply providing viewpoint information to media processing device 200. Therefore, although a delay occurs between media processing device 200 and user terminal 300, the processing load on user terminal 300 can be reduced.
[0154] In operation example 1, the media processing device 200 transmits the viewpoint information used in generating the specific content to the user terminal 300 in association with the sequence number added to the specific content. With this configuration, the user terminal 300 can grasp the viewpoint information and sequence number used in generating the specific content, and so can appropriately display the specific content even in cases where the viewpoint information used when the media processing device 200 generates the specific content differs from the viewpoint information used when the user terminal 300 displays the specific content.
[0155] In operation example 2, the media processing device 200 receives a content configuration and specific viewpoint information (recommended viewport information) from the transmitting device 100. With this configuration, the media processing device 200 can generate specific content based on the specific viewpoint information, and can appropriately display the specific content even in a case where a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed.
[0156] In operation example 2, the media processing device 200 generates specific content based on specific viewpoint information in response to a reset signal, even when the user terminal is the first user terminal 400. With this configuration, even in cases where the viewpoint position and line of sight direction become unclear in the three-dimensional space constructed by the scene description in the first user terminal 400 (where the user becomes lost in the three-dimensional space), the reset signal can return to specific content based on specific viewpoint information.
[0157] In operation example 3, the media processing device 200 receives quality information about each of two or more streams, the quality of which varies depending on the orientation of the 3D object based on viewpoint information, from the transmitting device 100 as quality information about a stream related to a 3D object included in specific content. With this configuration, based on the new knowledge that the quality of each face constituting a bounding box for a 3D object does not have to be uniform, it is possible to appropriately display a 3D object while suppressing transmission traffic.
[0158] In operation example 4, media processing device 200 receives importance information for each of two or more objects included in specific content from transmitting device 100. With this configuration, by introducing a mechanism for setting the importance of each object, such as a 360° video image or a 3D object, it is possible to appropriately display each object included in the specific content while suppressing transmission traffic.
[0159] In operation example 5, media processing device 200 receives from transmitting device 100 an information element that defines the range of movement of the user's viewpoint position in the three-dimensional space constituted by the specific content. With this configuration, specific content including content with freedom of viewpoint can be appropriately displayed without causing a breakdown in the specific content displayed on user terminal 300.
[0160] [Change Example 1] Modification 1 of the embodiment will be described below. Differences from the embodiment will be mainly described below.
[0161] In the first modification, a method for synchronizing the first content with the second content when a specific content includes both the first content and the second content will be described.
[0162] In the following, synchronization means that the presentation times of the first content (e.g., MPU) and the second content (file) are properly synchronized. Therefore, synchronization may include the presentation times of the 2D video and the 3D object being synchronized, and may include the presentation times of the audio and the 3D object being synchronized. Similarly, synchronization may include the presentation times of the 2D video and the 360° video being synchronized, and may include the presentation times of the audio and the 360° video being synchronized.
[0163] In the first method, a case will be described in which media processing device 200 synchronizes the first content and the second content based on first control information (MMT-SI). Media processing device 200 uses MMT-SI as an entry point to check whether or not there is a scene description (second content), and if there is a scene description, identifies the presentation time of specific content including the first content and the second content by using the MPU timestamp descriptor.
[0164] Specifically, as shown in FIG. 19, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 19), and therefore the 2D video and audio can be synchronized.
[0165] On the other hand, the presentation time of the first frame included in the scene description is specified by referring to the MPU timestamp descriptor included in the MMT-SI. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the second content. For example, in the case where the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1 / 30×n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".
[0166] In the first method, a case has been exemplified in which the scene description does not include the presentation time of the first frame included in the scene description, but the scene description may include the presentation time of the first frame included in the scene description.
[0167] In the second method, a case will be described in which media processing device 200 synchronizes the first content and the second content based on second control information (scene description). Media processing device 200 uses the scene description as an entry point to check whether or not MMT-SI (first content) is present, and if MMT-SI is present, identifies the presentation time of specific content including the first content and the second content based on the presentation time included in the scene description.
[0168] In such a case, the scene description includes absolute time information indicating the presentation time of the second content, which may be the presentation time of the first frame included in the scene description.
[0169] For example, the absolute time information may be generated using UTC as the reference time. The reference time may be the TAI or a time provided by a GPS. The reference time may be a time provided by an NTP server or a time provided by a PTP server. Furthermore, the absolute time information may be generated based on the same reference time as the MPU timestamp descriptor.
[0170] Furthermore, the scene description includes reference information for identifying the first content. The reference information may be information for identifying an MPU that constitutes the first content. In other words, the reference information is information for treating the first content (MPU) as an object included in the scene description.
[0171] Specifically, as shown in Fig. 20, the presentation time of the first frame included in the scene description is specified by the absolute time information included in the scene description. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the second content. For example, in the case where the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1 / 30 x n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".
[0172] On the other hand, since the 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 20), the 2D video and audio can be synchronized. Here, since the above-mentioned reference information is included in the scene description, the media processing device 200 can check whether or not there is a first content to be presented together with the second content, based on the reference information included in the scene description.
[0173] In the second method, synchronization between 2D video and audio is achieved based on the MPU timestamp descriptor included in MMT-SI, but in the first modification, synchronization between 2D video and audio may also be achieved based on information elements (absolute time information and reference information) included in the scene description. In such a case, at least the MPU timestamp descriptor included in MMT-SI may be omitted. Furthermore, MMT-SI itself may be omitted.
[0174] In addition, when the reference time (hereinafter, the first reference time) of the MPU timestamp descriptor included in the MMT-SI is different from the reference time (the second reference time) of the absolute time information included in the scene description, at least one of the first control information (MMT-SI) and the second control information (scene description) may include conversion information between the first reference time and the second reference time. For example, the MMT-SI may include an MPU timestamp descriptor expressed in the second reference time (e.g., a reference time other than UTC) in addition to the MPU timestamp descriptor expressed in the first reference time (e.g., UTC). The scene description may include absolute time information expressed in the first reference time (e.g., UTC) in addition to the absolute time information expressed in the second reference time (e.g., a reference time other than UTC).
[0175] In addition, the MPU timestamp descriptor included in the MMT-SI may be referred to as first absolute time information, and the absolute time information included in the scene description may be referred to as second absolute time information.
[0176] [Change Example 2] In the following, a second modification of the embodiment will be described. In the following, differences from the embodiment will be mainly described. Specifically, in the second modification, a scene description editing device used to edit a scene description that defines a specific content will be mainly described.
[0177] (Media processing device and scene description editing device) The following describes a media processing device and a scene description editing device according to Modification 2. Figure 21 is a block diagram showing a media processing device 200 and a scene description editing device 600 according to Modification 2.
[0178] First, the media processing device 200 includes a receiving unit 210, a renderer 220, and an encoding processing unit 230.
[0179] The receiving unit 210 receives the viewpoint information. In a second modification, the receiving unit 210 may receive the viewpoint information from the scene description editing device 600.
[0180] The renderer 220 generates specific content including at least the second content based on the viewpoint information. In the following, a case will be illustrated in which the specific content includes the first content in addition to the second content.
[0181] First, the renderer 220 generates a first content including 2D video and audio as a part of a specific content based on the first control information (MMT-SI). Viewpoint information is not required in generating the first content.
[0182] Here, in the embodiment described above, the media processing device 200 receives the first control information (MMT-SI), 2D video and audio from the transmitting device 100, but in modification 2, the media processing device 200 receives the first control information (MMT-SI), 2D video and audio from the scene description editing device 600.
[0183] Secondly, the renderer 220 generates a second content including a 360° video and a 3D object as a part of the specific content based on the second control information (scene description). Viewpoint information is used in generating the second content.
[0184] Here, in the embodiment described above, the media processing device 200 receives the second control information (scene description), the 360° video footage, and the 3D objects from the transmitting device 100, but in modification 2, the media processing device 200 receives the second control information (scene description), the 360° video footage, and the 3D objects from the scene description editing device 600.
[0185] The encoding processing unit 230 encodes the specific content generated by the renderer 220. In a second modification, the encoding processing unit 230 may transmit the specific content to the scene description editing device 600.
[0186] Secondly, the scene description editing device 600 includes a detection unit 610, a decoding processing unit 620, a renderer 630, an editing unit 640, and a database 650.
[0187] The detection unit 610 detects the user's viewpoint position and line of sight direction, similar to the detection unit 310 described above. The detection unit 610 may include an acceleration sensor or a GPS sensor. The detection unit 610 may include a user I / F (e.g., a touch sensor, a keyboard, a mouse, a controller, etc.) that is manually input by the user. The detection unit 610 may transmit viewpoint information (viewpoint position and line of sight direction) to the media processing device 200. The detection unit 610 may output the viewpoint information (viewport) to the renderer 630.
[0188] The decoding processing unit 620, like the decoding processing unit 320 described above, decodes the specific content received from the media processing device 200. The decoding processing unit 620 may decode the presentation time received from the media processing device 200. The decoding processing unit 620 may output the specific content to the renderer 630, or may output the presentation time to the renderer 630.
[0189] The renderer 630, like the above-described renderer 330, outputs the specific content decoded by the decoding processing unit 620. The renderer 630 may output the specific content based on the presentation time decoded by the decoding processing unit 620. For example, the renderer 630 may output the video content included in the specific content to a display and output the audio content included in the specific content to a speaker.
[0190] In the second modification, the specific content output from the renderer 630 is used for editing the scene description or the like, and may therefore be considered as a preview of the specific content.
[0191] The editing unit 640 is a component used to edit a scene description that defines the configuration of content (second content) having freedom of viewpoint. The editing unit 640 may transmit the scene description to the media processing device 200. When the scene description is updated, the editing unit 640 may transmit the updated scene description to the media processing device 200. Updating the scene description may be interpreted as editing the scene description. The editing unit 640 may include a user I / F (e.g., a touch sensor, a keyboard, a mouse, a controller, etc.) for editing the scene description.
[0192] As described above, the scene description is generated for each file, and includes information for identifying the 360° video and the 3D object for each frame. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, and an information element (scale_object) indicating the size of the 3D object in the frame.
[0193] The database 650 stores media such as 3D objects, 2D images, audio, 360° images, etc. The database 650 may transmit required media files to the media processing device 200 in response to a transmission request or a read request from the media processing device 200.
[0194] (UI of scene description editor) The following describes an example of a UI (User Interface) of the scene description editing device according to Modification 2. Fig. 22 is a diagram showing an example of the UI of the scene description editing device 600 according to Modification 2. The UI of the scene description editing device 600 may be considered as an editing screen for editing the scene description.
[0195] As shown in FIG. 22, the editing interface (UI) may include a media list, a timeline, a preview, a scene description, and the like.
[0196] The media list is an area that displays a list of media such as 3D objects, 2D video, audio, and 360° video specified by the creator of a particular content (hereinafter simply referred to as the creator).
[0197] The timeline is an area where elements that make up a scene are selected from a media list by the creator's operation, and the selected elements are arranged on the timeline. The timeline includes an indicator (such as a line) that indicates the time selected by the creator on the timeline, and the scene at the time indicated by the indicator may be displayed in the preview as a 2D image of the specific content.
[0198] The preview is an area where specific content is displayed according to viewpoint information (viewpoint position and viewpoint direction) specified by the creator's operation. The viewpoint information may be detected by the above-mentioned detection unit 610. The creator may edit the scene description while checking the specific content displayed in the preview. The preview may be displayed by a head-mounted display used by the creator. In addition, two or more previews may be provided, such as displaying the same scene at the same time from different viewpoints, such as looking downward from the sky or looking east to west. By displaying the same scene from multiple different viewpoints, the scene description can be edited while easily checking the arrangement of objects in three-dimensional space.
[0199] The scene description is an area where the scene description edited by the creator is displayed. The scene description may include specific viewpoint information (recommended viewport information) described in Operation Example 2 (see, for example, FIG. 7). The scene description may include quality information and placement information described in Operation Example 3 (see, for example, FIG. 10). The scene description may include importance information described in Operation Example 4 (see, for example, FIG. 14). The scene description may include information elements that define the moving range of the user's viewpoint position described in Operation Example 5 (see, for example, FIG. 17 and FIG. 18).
[0200] (Example of operation) An editing operation using the scene description editing device 600 will now be described.
[0201] First, when the creator starts editing the scene description, the scene description editing device 600 may notify the media processing device 200 of the start of editing the scene description by RTSP SETUP. The RTSP SETUP may include the IP address and listening port number of the scene description editing device 600, a request to operate in edit mode, and the like.
[0202] Secondly, the viewpoint information (viewpoint position and viewpoint direction) can be changed by an operation by the creator. When the viewpoint information is changed by an operation by the creator, the changed viewpoint information is transmitted from the scene description editing device 600 to the media processing device 200. This mechanism is similar to the mechanism for transmitting viewpoint information from the user terminal 300 to the media processing device 200.
[0203] Third, parameters such as the position, size, and orientation of elements such as 3D objects can be changed by an operation (editing) by the creator. When parameters are changed by an operation by the creator, the changed parameters are reflected in the scene description. The changed scene description is transmitted from the scene description editing device 600 to the media processing device 200 and rendered by the renderer 220 of the media processing device 200, and the rendered specific content is transmitted from the media processing device 200 to the scene description editing device 600. That is, the specific content displayed in the preview is updated in response to the change in parameters. The mechanism by which the specific content based on the scene description is transmitted from the media processing device 200 to the scene description editing device 600 is similar to the mechanism by which the specific content is transmitted from the media processing device 200 to the user terminal 300.
[0204] Fourth, by the creator selecting an icon 710 ("recommended view" in FIG. 22) included in the editing screen (UI), a specific content corresponding to the specific viewpoint information (recommended viewport information) described in operation example 2 may be displayed in the preview.
[0205] Fifth, viewpoint information corresponding to a specific content displayed in the preview may be recorded as the specific viewpoint information (recommended viewport information) described in operation example 2 by the creator's operation of selecting an icon 720 ("recommended view record" in FIG. 22) included in the editing screen (UI). The specific viewpoint information (recommended viewport information) recorded by the icon 720 may be reflected in the scene description. For example, viewpoint information corresponding to a specific content displayed in the preview may be operated by a head mounted display, and the viewpoint information operated by the head mounted display may be recorded as the specific viewpoint information (recommended viewport information) by selecting the icon 720. That is, the specific viewpoint information is generated according to the operation of the head mounted display used by the creator of the specific content.
[0206] Sixth, the information element (for example, "Viewing_space" shown in FIG. 18) that defines the movement range of the user's viewpoint position described in Operation Example 5 may be specified by the creator's operation (such as a mouse operation). The information element ("Viewing_space") that defines the specified movement range may be reflected in the scene description. It may be possible to switch whether or not to display the information element ("Viewing_space") that defines the movement range (display or non-display).
[0207] In Modification 2, the scene description editing device 600 may be implemented as a separate entity from the media processing device 200, and may be connected to the media processing device 200 via a network. For example, the media processing device 200 may be implemented as Software as a Service (SaaS) on the cloud. However, Modification 2 is not limited to this. For example, the scene description editing device 600 may be implemented as a single device (computer, hardware) integrated with the media processing device 200. In such a case, the encoding processing unit 230 and the decoding processing unit 620 described above may be omitted. Furthermore, transmission may be read as output, and reception may be read as acquisition.
[0208] (Action and Effects) In Modification 2, the scene description editing device 600 acquires the specific content generated by the media processing device 200 based on the scene description and the viewpoint information. With this configuration, assuming a case in which the specific content is generated (rendered) by the media processing device 200 and then the generated specific content is transmitted from the media processing device 200 to the user terminal 300, the scene description editing device 600 can acquire the specific content from the media processing device 200 in a manner similar to that of the user terminal 300. Therefore, it is possible to appropriately edit the scene description while checking the specific content that is expected to be displayed on the user terminal 300.
[0209] [Change Example 3] The following describes Modification Example 3 of the embodiment. The following mainly describes the differences from Modification Example 2. Specifically, in Modification Example 3, similar to Modification Example 2, the following mainly describes a scene description editing device used to edit a scene description that defines a specific content.
[0210] (Media processing device and scene description editing device) The following describes a media processing device and a scene description editing device according to Modification 3. Figure 23 is a block diagram showing a media processing device 200 and a scene description editing device 600 according to Modification 3.
[0211] First, the media processing device 200 includes a receiving unit 210, a renderer 220, and an encoding processing unit 230.
[0212] The receiving unit 210 receives viewpoint information in the same manner as in Modification Example 2. In Modification Example 3, the receiving unit 210 may receive two or more pieces of viewpoint information (1 to N) from the scene description editing device 600. The two or more pieces of viewpoint information may be viewpoint information that differs in at least one of the viewpoint position and the line of sight direction. Here, N is an integer of 2 or more.
[0213] For example, the reception unit 210 receives two or more pieces of viewpoint information (1 to N) by a VP message shown in FIG.
[0214] The renderer 220 generates specific content including at least the second content based on viewpoint information, similar to the modified example 2. In the following, a case where the specific content includes the first content in addition to the second content will be illustrated. In the modified example 3, the renderer 220 generates two or more specific contents (1 to N) based on two or more pieces of viewpoint information (1 to N), respectively.
[0215] The renderer 220 may transmit two or more pieces of viewpoint information (1 to N) used to generate two or more specific contents (1 to N), respectively, to the scene description editing device 600. Note that N is an integer of 2 or more.
[0216] It should be noted here that the two or more specific contents (1 to N) are contents having different viewpoint information at the same time.
[0217] For example, the renderer 220 may identify viewpoint information corresponding to each of two or more specific contents based on the source port number of the VP message corresponding to each of the two or more viewpoint information (1 to N). Alternatively, when the IP address of the VP message is IPv6, the renderer 220 may identify viewpoint information corresponding to each of two or more specific contents based on the IP address of the VP message corresponding to each of the two or more viewpoint information (1 to N).
[0218] For example, assuming that the frame rate of each of the N specific contents is 30 fps, the renderer 220 generates N×30 fps specific contents based on each of two or more pieces of viewpoint information (1 to N). In detail, the renderer 220 may generate N×30 fps specific contents (previews) in the order of frame #1 of preview #1→frame #1 of preview #2→···frame #1 of preview #N→frame #2 of preview #1→frame #2 of preview #2→···frame #2 of preview #N→···frame #30 of preview #1→frame #30 of preview #2→···frame #30 of preview #N.
[0219] The encoding processing unit 230 encodes the specific content generated by the renderer 220. In a third modification, the encoding processing unit 230 may transmit, to the scene description editing device 600, two or more specific contents (1 to N) corresponding to two or more pieces of viewpoint information (1 to N), respectively.
[0220] Secondly, the scene description editing device 600 includes a detection unit 610, a decoding processing unit 620, a renderer 630, an editing unit 640, and a database 650.
[0221] The detection unit 610 detects the user's viewpoint position and line of sight (i.e., viewpoint information) in the same manner as in the second modification. In the third modification, the detection unit 610 detects two or more pieces of viewpoint information (1 to N). For example, the detection unit 610 detects the following: detection unit 6101 to detection unit 610 N The detection unit 6101 to the detection unit 610 N Alternatively, two or more pieces of viewpoint information (1 to N) may be output to the media processing device 200.
[0222] As in Modification 2, the decoding processing unit 620 decodes the specific content received from the media processing device 200. In Modification 3, the decoding processing unit 620 may decode two or more specific contents (1 to N).
[0223] The renderer 630 outputs the specific content decoded by the decode processing unit 620, similarly to the second modification. In the third modification, the renderer 630 includes the renderers 6301 to 6302. N The renderer 6301 to the renderer 630 N Alternatively, two or more specific contents (1 to N) corresponding to two or more pieces of viewpoint information (1 to N) may be output.
[0224] For example, the renderer 6301 outputs specific content (1) corresponding to viewpoint information (1), . . . , and outputs specific content (N) corresponding to viewpoint information (N).
[0225] As described above, the two or more specific contents (1 to N) are contents having different viewpoint information at the same time. Therefore, it should be noted that the two or more specific contents (1 to N) need to be output at the same time.
[0226] Here, the specific content output from the renderer 630 may be considered to be a preview of the specific content, since it is used for editing the scene description, etc., as in the modified example 2. In the modified example 3, the two or more specific contents (1 to N) output from the renderer 630 may be considered to be previews corresponding to the respective pieces of viewpoint information (1 to N).
[0227] As in the second modification, the editing unit 640 is configured to be used for editing a scene description that defines the configuration of a content (second content) having freedom of viewpoint.
[0228] In the third modification, the detection unit 610 and the editing unit 640 constitute an output unit that outputs a scene description that defines the configuration of content with viewpoint freedom and two or more pieces of viewpoint information to the media processing device 200. The decoding processing unit 620 constitutes an acquisition unit that acquires two or more specific contents that include at least the content and that have been generated by the media processing device 200 based on the scene description and each of the two or more pieces of viewpoint information.
[0229] (UI of scene description editor) The following describes an example of a UI (User Interface) of the scene description editing device according to Modification 3. Fig. 24 is a diagram showing an example of the UI of the scene description editing device 600 according to Modification 3. The UI of the scene description editing device 600 may be considered as an editing screen for editing the scene description.
[0230] 24, the editing screen (UI) may include a media list, a timeline, a preview, a scene description, etc., similar to the modified example 2. Since the media list, the timeline, the scene description, etc. are the same as the modified example 2, the preview will be mainly described below.
[0231] Similar to modification example 2, the preview is an area where a specific content is displayed according to viewpoint information (viewpoint position and viewpoint direction) specified by the creator's operation. In modification example 3, the preview includes two or more previews corresponding to two or more viewpoint information (1 to N), respectively. Each of the two or more previews is one of two or more specific contents (1 to N).
[0232] As described above, the two or more specific contents (1 to N) are contents having different viewpoint information at the same time, and therefore the two or more previews are previews having different viewpoint information at the same time.
[0233] Although not particularly limited, when an operation such as "open a new preview" is performed on the editing screen (UI) shown in FIG. 22, the number of previews on the editing screen (UI) may increase by one.
[0234] (2 or more specific content output methods) An example of a method for outputting two or more specific contents according to Modification Example 3 will be described below. Figs. 25 to 27 are diagrams showing an example of a method for outputting two or more specific contents according to Modification Example 3. As a method for outputting two or more specific contents, the following options are considered.
[0235] In option 1, the media processing device 200 transmits a rectangular video including two or more specific contents corresponding to the same time to the scene description editing device 600, as shown in Figure 25. In other words, the decoding processing unit 620 described above obtains a rectangular video including two or more specific contents corresponding to the same time.
[0236] For example, as shown in Fig. 25, the rectangular image may include two or more specific contents arranged horizontally. However, the third modification example is not limited to this. The rectangular image may include two or more specific contents arranged vertically.
[0237] The media processing device 200 (for example, the encoding processing unit 230) transmits placement information indicating positions where two or more specific contents are placed in a rectangular video to the scene description editing device 600. As the placement information, an identifier for identifying a tile or slice, which is a unit of compression encoding, may be used.
[0238] In option 1, the scene description editing device 600 (for example, the decoding processing unit 620) extracts each of the two or more specific contents from the rectangular video based on the arrangement information, and outputs each of the two or more specific contents to the renderer 630 (renderer 6301 to renderer 630 N ) may be output.
[0239] In option 2, the media processing device 200 outputs a rectangular video including two or more specific contents corresponding to the same time to the scene description editing device 600, as shown in Fig. 26. In other words, the decoding processing unit 620 described above obtains a rectangular video including two or more specific contents corresponding to the same time.
[0240] For example, a rectangular image may include two or more specific contents while maintaining a specific resolution, as shown in Fig. 26. The specific resolution may be a commonly used resolution such as 3840 x 2160 pixels, 4096 x 2160 pixels, 7680 x 4320 pixels, etc.
[0241] Under such a premise, depending on the size or number of two or more specific contents included in the rectangular video, there may be a case where a blank area other than the two or more specific contents is generated in the rectangular video. Such a blank area is treated as a padding area. In other words, the rectangular video includes a padding area in addition to the two or more specific contents.
[0242] In the case where the aspect ratio of the rectangular image is the same as the aspect ratio of each specific content and the rectangular image contains three specific contents, the rectangular image may include a padding area having a size similar to the specific content, as shown in FIG. 26.
[0243] The media processing device 200 (for example, the encoding processing unit 230) transmits placement information indicating positions where two or more specific contents are placed in a rectangular video to the scene description editing device 600. As the placement information, an identifier for identifying a tile or slice, which is a unit of compression encoding, may be used.
[0244] In option 2, the scene description editing device 600 (for example, the decoding processing unit 620) extracts each of the two or more specific contents from the rectangular video based on the arrangement information, and outputs each of the two or more specific contents to the renderer 630 (renderer 6301 to renderer 630 N ) may be output.
[0245] In option 3, media processing device 200 transmits two or more specific contents corresponding to the same time consecutively on the time axis, as shown in Figure 27. In other words, decoding processing unit 620 described above acquires two or more specific contents corresponding to the same time consecutively on the time axis.
[0246] For example, assuming that the frame rate of each of the N specific contents is 30 fps, the media processing device 200 (renderer 220) outputs the N×30 fps specific contents continuously on the time axis. The media processing device 200 (e.g., the encoding processing unit 230) may transmit order information indicating the order in which the N specific contents are to be transmitted to the scene description editing device 600. Alternatively, the media processing device 200 (e.g., the encoding processing unit 230) may transmit the N specific contents based on a predetermined rule. The predetermined rule may be a rule that the specific contents are transmitted in ascending order of the numbers (e.g., preview numbers) assigned to the specific contents.
[0247] In option 3, the scene description editing device 600 (for example, the decoding processing unit 620) extracts each of two or more specific contents based on the order information or a predetermined rule, and outputs each of the two or more specific contents to a renderer 630 (renderer 6301 to renderer 6302). N ) may be output.
[0248] Note that in option 3, a case may be assumed in which compression encoding of two or more specific contents is not performed. In such a case, the media processing device 200 may transmit each of the two or more specific contents to a transmission port number corresponding to each of the two or more pieces of viewpoint information as a destination. The scene description editing device 600 transmits each of the two or more specific contents to a renderer 630 (renderer 6301 to renderer 6302) based on the transmission port number used as the destination. N ) may be output.
[0249] (others) As explained in the embodiment, the media processing device 200 (renderer 220) transmits specific content to the user terminal 300 based on one piece of viewpoint information. On the other hand, in modification example 3, the media processing device 200 (renderer 220) transmits two or more pieces of specific content corresponding to the same time to the scene description editing device 600 based on each of two or more pieces of viewpoint information.
[0250] Therefore, assuming that the processing capability of the renderer 220 (e.g., the number of frames that can be generated per second (fps)) is constant, the frame rate of each of two or more specific contents corresponding to the same time may be lower than the frame rate of the specific content transmitted to the user terminal 300.
[0251] For example, assuming that the frame rate of the specific content transmitted to the user terminal 300 (i.e., the processing capacity of the renderer 220) is 120 fps, if there are four specific contents (previews) corresponding to the same time, the frame rate of each of the four specific contents may be 30 fps. In other words, the frame rate of the specific content transmitted to the user terminal 300 may be the sum of the frame rates of two or more specific contents (previews) corresponding to the same time.
[0252] (Action and Effects) In the third modification, the scene description editing device 600 acquires two or more specific contents (previews) generated by the media processing device 200 based on the scene description and each of the two or more pieces of viewpoint information. With this configuration, assuming a case in which the specific contents are generated (rendered) by the media processing device 200 and then the generated specific contents are transmitted from the media processing device 200 to the user terminal 300, the scene description editing device 600 can acquire two or more specific contents having different viewpoint information from the media processing device 200 in a similar manner to the user terminal 300. Therefore, for the specific contents expected to be displayed on the user terminal 300, it is possible to appropriately edit the scene description while checking two or more specific contents (previews) corresponding to the two or more pieces of viewpoint information.
[0253] In the third modification, the frame rate of each of the two or more specific contents corresponding to the same time is lower than the frame rate of the specific content transmitted to the user terminal 300. With this configuration, two or more specific contents corresponding to the same time can be generated by a single renderer 220 provided in the media processing device 200.
[0254] In Modification 3, the scene description editing device 600 acquires a rectangular image including two or more specific contents corresponding to the same time (Option 1 and Option 2 described above). With this configuration, even if the two or more specific contents are not associated with a presentation time (e.g., a timestamp), the scene description editing device 600 can easily synchronize the presentation times of two or more specific contents corresponding to the same time.
[0255] In Modification Example 3, the rectangular video includes a padding area in addition to two or more specific contents (Option 2 described above). With this configuration, it is possible for the scene description editing device 600 to handle rectangular videos having a specific resolution that is commonly used, without providing the scene description editing device 600 with a renderer 630 that requires special processing.
[0256] In Modification 3, the scene description editing device 600 continuously acquires the two or more specific contents corresponding to the same time on the time axis (Option 3 described above). With this configuration, even if the two or more specific contents are not associated with presentation times (e.g., timestamps), it is easy for the scene description editing device 600 to synchronize the presentation times of the two or more specific contents corresponding to the same time.
[0257] [Other embodiments] Although the present invention has been described by the above disclosure, the description and drawings forming a part of this disclosure should not be understood as limiting the present invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art.
[0258] In the above disclosure, the case where the specific content includes both the first content and the second content has been exemplified, but the above disclosure is not limited thereto. The specific content may include at least the second content.
[0259] Although not specifically mentioned in the above disclosure, terms related to MMT may be interpreted based on the contents defined in ISO / IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc.
[0260] In the above disclosure, the MPU timestamp descriptor is exemplified as the first absolute time information included in the MMT-SI. However, the above disclosure is not limited to this. The first absolute time information included in the MMT-SI may be an MPU extended timestamp descriptor.
[0261] Although not specifically mentioned in the above disclosure, media processing device 200 may request a portion of the second content from transmitting device 100 as necessary. This configuration makes it possible to conserve bandwidth associated with the transmission of the second content and to prevent an increase in the processing load on media processing device 200.
[0262] In the above disclosure, MMTP is exemplified as the transmission method of the first content. However, the above disclosure is not limited thereto. The transmission method of the first content may be a method conforming to ISO / IEC 23009-1 (hereinafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)). In such a case, the first control information may be MPD (Media Presentation Description). That is, in the above disclosure, MMT-SI may be read as MPD.
[0263] Although not specifically mentioned in the above disclosure, "obtain" may be read as "receive."
[0264] Although not particularly limited, the second operational example may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a configuration of content having freedom of viewpoint, and the transmitting unit transmits specific viewpoint information used to generate specific content including at least the content. The receiving device includes a receiving unit that receives a configuration of content having freedom of viewpoint, and the receiving unit receives the specific viewpoint information used to generate specific content including at least the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.
[0265] Although not particularly limited, the operation example 3 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a configuration of content having freedom of viewpoint, and the transmitting unit transmits quality information of a stream related to a three-dimensional object included in specific content including at least the content, and the quality information includes quality information related to each of two or more streams whose quality differs depending on the orientation of the three-dimensional object. The receiving device includes a receiving unit that receives a configuration of content having freedom of viewpoint, and the receiving unit receives quality information of a stream related to a three-dimensional object included in specific content including at least the content, and the quality information includes quality information related to each of two or more streams whose quality differs depending on the orientation of the three-dimensional object. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.
[0266] Although not particularly limited, the fourth operational example may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a configuration of content having freedom of viewpoint, and the transmitting unit transmits importance information regarding each of two or more objects included in specific content that at least includes the content. The receiving device includes a receiving unit that receives a configuration of content having freedom of viewpoint, and the receiving unit receives importance information regarding each of two or more objects included in specific content that at least includes the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.
[0267] Although not particularly limited, the operation example 4 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a configuration of content having freedom of viewpoint, and the transmitting unit transmits information elements that define a movement range of a user's viewpoint position in a three-dimensional space configured by specific content including at least the content. The receiving device includes a receiving unit that receives a configuration of content having freedom of viewpoint, and the receiving unit receives information elements that define a movement range of a user's viewpoint position in a three-dimensional space configured by specific content including at least the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.
[0268] Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the transmitting device 100, the media processing device 200, the user terminal 300, and the scene description editing device 600. The program may be recorded on a computer-readable medium. Using the computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.
[0269] Alternatively, a chip may be provided that is configured by a memory that stores programs for executing the processes performed by the transmitting device 100, the media processing device 200, the user terminal 300, and the scene description editing device 600, and a processor that executes the programs stored in the memory.
[0270] (Additional Note) The above disclosure may be expressed as follows:
[0271] A first feature is a scene description editing device comprising: an output unit that outputs a scene description and two or more pieces of viewpoint information that define a structure of content having viewpoint freedom to a media processing device; and an acquisition unit that acquires two or more pieces of specific content that include at least the content, the specific content being generated by the media processing device based on the scene description and each of the two or more pieces of viewpoint information.
[0272] A second feature is a scene description editing device in which, in the first feature, the frame rate of each of the two or more specific contents generated by the media processing device is lower than the frame rate of the specific content transmitted to a user terminal used by a viewer of the specific content.
[0273] A third feature is the scene description editing device according to the first or second feature, wherein the acquisition unit acquires a rectangular video including the two or more specific contents corresponding to the same time.
[0274] A fourth feature is the scene description editing device according to the third feature, wherein the rectangular image includes a padding area in addition to the two or more specific contents.
[0275] A fifth feature is the scene description editing device according to the first or second feature, wherein the acquisition unit continuously acquires each of the two or more specific contents corresponding to the same time on a time axis.
[0276] A sixth feature is a program causing a computer to execute the steps of: outputting, to a media processing device, a scene description and two or more pieces of viewpoint information that define a structure of content having viewpoint freedom; and acquiring, by the media processing device, two or more pieces of specific content that include at least the content, the specific content being generated by the media processing device based on the scene description and each of the two or more pieces of viewpoint information. [Explanation of symbols]
[0277] 10...transmission system, 100...transmitting device, 200...media processing device, 210...receiving unit, 220...renderer, 230...encoding processing unit, 260...selecting unit, 270...selecting unit, 300...user terminal, 310...detecting unit, 320...decoding processing unit, 330...renderer, 400...first user terminal, 500...second user terminal, 600...scene description editing device, 610...detecting unit, 620...decoding processing unit, 630...renderer, 640...editing unit, 650...database
Claims
1. an output unit that outputs a scene description that defines a configuration of content having freedom of viewpoint and two or more pieces of viewpoint information to a media processing device; a scene description editing device comprising: an acquisition unit that acquires two or more specific contents including at least the content, the specific contents being generated by the media processing device based on the scene description and each of the two or more viewpoint information.
2. The scene description editing device according to claim 1 , wherein a frame rate of each of the two or more specific contents generated by the media processing device is lower than a frame rate of the specific content transmitted to a user terminal used by a viewer of the specific content.
3. The scene description editing device according to claim 1 , wherein the acquisition unit acquires a rectangular image including the two or more specific contents corresponding to the same time.
4. The scene description editing device according to claim 3 , wherein the rectangular image includes a padding area in addition to the two or more specific contents.
5. The scene description editing device according to claim 1 , wherein the acquisition unit acquires each of the two or more specific contents corresponding to the same time continuously on a time axis.
6. Step A of outputting a scene description defining a configuration of content having freedom of viewpoint and two or more pieces of viewpoint information to a media processing device; and B. acquiring two or more specific contents including at least the content, the specific contents being generated by the media processing device based on the scene description and each of the two or more viewpoint information.
