Media processing device, transmitting device and receiving device

Abstract

This media processing device is provided with: a renderer that generates, on the basis of a configuration of content having a freedom of viewpoint, and specific viewpoint information, a specific content including at least said content; and an output unit that outputs the specific content generated by the renderer to a user terminal.

Description

[Technical Field]

[0001] The present disclosure relates to a media processing device, a transmitting device, and a receiving device. [Background technology]

[0002] Conventionally, mechanisms for transmitting content such as 360-degree video and 3D objects have been proposed (for example, Non-Patent Document 1). Known mechanisms include 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range of head movement of a seated user, and 6DoF, which involves viewpoint movement within the range of free movement of a user. In such mechanisms, the positional relationship between the 360-degree video and the 3D objects is indicated by a scene description. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] 3GPP TR 26.928 V16.1.0 December 2020 Summary of the Invention

[0004] One aspect of the disclosure is a media processing device that includes a renderer that generates specific content that includes at least the content based on a content structure having viewpoint freedom and specific viewpoint information, and an output unit that outputs the specific content generated by the renderer to the user terminal.

[0005] One aspect of the disclosure is a transmitting device that includes a transmitting unit that transmits a content configuration having viewpoint freedom, and the transmitting unit transmits specific viewpoint information used to generate specific content that includes at least the content.

[0006] One aspect of the disclosure is a receiving device that includes a receiving unit that receives a content configuration having viewpoint freedom, and the receiving unit receives specific viewpoint information used to generate specific content that includes at least the content. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing a transmission system 10 according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating the second content according to the embodiment. [Figure 4] FIG. 4 is a diagram showing a method for viewing a specific content according to the embodiment. [Figure 5] FIG. 5 is a diagram for explaining the first operation example. [Figure 6] FIG. 6 is a diagram for explaining the second operation example. [Figure 7] FIG. 7 is a diagram for explaining the second operation example. [Figure 8] FIG. 8 is a diagram for explaining the third operation example. [Figure 9] FIG. 9 is a diagram for explaining the third operation example. [Figure 10] FIG. 10 is a diagram for explaining the third operation example. [Figure 11] FIG. 11 is a diagram for explaining the third operation example. [Figure 12] FIG. 12 is a diagram for explaining the fourth operation example. [Figure 13] FIG. 13 is a diagram for explaining the fourth operation example. [Figure 14] FIG. 14 is a diagram for explaining the fourth operation example. [Figure 15] FIG. 15 is a diagram for explaining the fifth operation example. [Figure 16] FIG. 16 is a diagram for explaining the fifth operation example. [Figure 17]FIG. 17 is a diagram for explaining the fifth operation example. [Figure 18] FIG. 18 is a diagram for explaining the fifth operation example. [Figure 19] FIG. 19 is a diagram for explaining the first method according to the first modification. [Figure 20] FIG. 20 is a diagram for explaining the second method according to the first modification. DETAILED DESCRIPTION OF THE INVENTION

[0008] Next, an embodiment of the present invention will be described. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, it should be noted that the drawings are schematic and the ratios of the dimensions may differ from those of the actual parts.

[0009] Therefore, specific dimensions should be determined with reference to the following explanation. Of course, the dimensional relationships and ratios may differ between the drawings.

[0010] [Disclosure Summary] The media processing device according to the disclosure summary includes a renderer that generates specific content that includes at least the content based on a content structure having viewpoint freedom and specific viewpoint information, and an output unit that outputs the specific content generated by the renderer to the user terminal.

[0011] In the summary of the disclosure, a media processing device generates specific content based on a content configuration and specific viewpoint information. With this configuration, the media processing device can generate the specific content based on the specific viewpoint information, and can appropriately display the specific content even in a case where a first user terminal that feeds back viewpoint information and a second user terminal that does not feed back viewpoint information are mixed.

[0012] It should be noted that the specific content generated by the media processing device is generated based on viewpoint information, and the video included in the specific content can be treated as 2D video on the user terminal side. The viewpoint information used to generate the specific content may include specific viewpoint information or viewpoint information received from the user terminal.

[0013] [Embodiment] (Transmission System) A transmission system according to an embodiment will be described below. Fig. 1 shows a transmission system 10 according to an embodiment. As shown in Fig. 1, the digital wireless transmission system includes a transmitting device 100, a media processing device 200, and a user terminal 300.

[0014] In the embodiment, transmitting device 100 transmits a first content that does not have viewpoint freedom and a second content that has viewpoint freedom to media processing device 200. Furthermore, transmitting device 100 transmits first control information associated with the first content and second control information associated with the second content to media processing device 200.

[0015] The first content may include at least one of 2D video and audio. The first content and the first control information may be transmitted in a first format. The first format may be a format compliant with ISO / IEC 23008-1 (hereinafter, MMT (MPEG Media Transport)). In the following, a case will be exemplified in which the first format is MMTP (MMT Protocol) compliant with MMT. The first control information may be referred to as MMT-SI (Signaling Information).

[0016] The second content may include 360-degree video and 3D objects. The second content and second control information may be transmitted in a second manner. Alternatively, they may be transmitted using a protocol such as HTTP (Hyper Text Transfer Protocol). The second content may conform to 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range of a user's head movement while sitting, or 6DoF, which involves viewpoint movement within the range of a user's free movement. Since the second content has viewpoint freedom, it may include two or more 360-degree video and two or more 3D objects at the same time (frame). The second control information may be referred to as a scene description.

[0017] Here, the second control information may be transmitted in the above-described first method. That is, the second control information may be transmitted in the same first method (e.g., MMTP) as the first control information. Alternatively, the second control information may be transmitted in a protocol such as HTTP.

[0018] The transmission from transmitting device 100 to media processing device 200 is not particularly limited, but may be transmission using satellite broadcasting, transmission using the Internet network, or transmission using a mobile communications network.

[0019] Although not particularly limited, the transmission system may be a digital wireless transmission system, and the digital wireless transmission system may be a system used for 4K and 8K satellite broadcasting.

[0020] Based on the viewpoint information received from user terminal 300, media processing device 200 generates specific content that includes at least the second content described above, and transmits the generated specific content to user terminal 300. Although not particularly limited, transmission of the specific content may be via the Internet network or a mobile communication network.

[0021] The user terminal 300 may be a user terminal such as a smartphone, a tablet terminal, or a head-mounted display. As shown in Fig. 1, two or more user terminals 300 may be provided as the user terminal 300. In other words, two or more user terminals 300 may request the media processing device 200 to generate specific content. Each user terminal 300 may transmit different viewpoint information to the media processing device 200.

[0022] (Media processing devices and user terminals) The media processing device and user terminal according to the embodiment are described below. Figure 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to the embodiment.

[0023] First, the media processing device 200 includes a receiving unit 210, a renderer 220, and an encoding processing unit 230.

[0024] The reception unit 210 receives viewpoint information. In the embodiment, the reception unit 210 constitutes a receiving unit that receives the viewpoint information from the user terminal 300. The viewpoint information includes an information element indicating the viewpoint position of the user of the user terminal 300 and an information element indicating the line of sight of the user of the user terminal 300.

[0025] The renderer 220 generates specific content including at least the second content based on the viewpoint information. Since the specific content is generated based on the viewpoint information, it may include one 360° video and one 3D object at the same time (frame). In the following, an example will be given in which the specific content includes the first content in addition to the second content.

[0026] 2, the renderer 220 generates a first content including 2D video and audio as part of a specific content based on the first control information (MMT-SI). Viewpoint information is not required in generating the first content.

[0027] Specifically, the renderer 220 acquires the 2D video, audio, and MMT-SI in the form of MMTP packets in which the 2D video, audio, and MMT-SI are packetized.

[0028] For example, an MMTP packet is stored in an IP (Internet Protocol) packet, which may be transmitted using UDP (User Datagram Protocol) or TCP (Transmission Control Protocol).

[0029] Here, the first content is processed in units (hereinafter referred to as MPUs; Media Processing Units) that are separated by a fixed time width. An MPU includes one or more access units. An access unit may also be treated as an MFU (Media Fragment Unit). An MFU related to 2D video may be referred to as a NAL (Network Abstraction Layer) unit, and an MFU related to audio may be referred to as an MHAS (MPEG-H 3D Audio Stream) packet.

[0030] The MMT-SI includes a PA (Package Access) message, and the PA message includes an MPT (MMT Package Table) that lists the first content. Furthermore, the MMT-SI includes an MPU timestamp descriptor that indicates the presentation time of the first content. The MPU timestamp descriptor may indicate the presentation time of the MPU, i.e., the time of the first access unit to be presented in the MPU.

[0031] The MPU timestamp descriptor may be generated using UTC (Coordinated Universal Time) as the reference time. The reference time may be TAI (International Atomic Time) or a time provided by a GPS (Global Positioning System). The reference time may be a time provided by an NTP (Network Time Protocol) server or a PTP (Precision Time Protocol) server.

[0032] Second, the renderer 220 generates, as part of the specific content, a second content including a 360-degree video and a 3D object based on the second control information (scene description). Viewpoint information is used in generating the second content.

[0033] Specifically, the renderer 220 may acquire the scene description in the form of an MMTP packet in which the scene description is packetized. There are no particular limitations on the method for acquiring the 360° video and the 3D objects.

[0034] The 360° video may be converted into 2D video using a projective transformation such as ERP (Equirectangular Projection) or cube mapping. Metadata indicating the type of projective transformation applied to the 360° video may be added. The 3D object may be encoded in a mesh format. For mesh format encoding, ISO / IEC 14496-16 "Animation framework extension (AFX)" may be used. The 3D object may be encoded in a point cloud format. For point cloud format encoding, ISO / IEC 23090-5 "Video-based Point Cloud Compression" may be used.

[0035] Here, the second content is collected into a single file in units separated by a fixed time interval. The fixed time interval may be 500 ms. For example, if the frame rate is 60 fps (frames per second), one file contains 30 frames.

[0036] The scene description is generated for each file and includes information for specifying the 360-degree video and 3D objects for each frame. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, and an information element (scale_object) indicating the size of the 3D object in the frame.

[0037] Third, the renderer 220 outputs the specific content including the first content and the second content to the encoding processing unit 230. The renderer 220 may output the presentation time of the specific content to the encoding processing unit 230 together with the specific content.

[0038] Here, the presentation time of the specific content may be corrected based on the delay time between media processing device 200 and user terminal 300. Specifically, renderer 220 may calculate the presentation time (T' = T + ΔT) of the specific content to be provided from media processing device 200 to user terminal 300 based on the presentation time (T) and delay time (ΔT) of the specific content provided from transmitting device 100 to media processing device 200. The delay time (ΔT) may be a value that is predetermined in media processing device 200, or may be a value that differs for each user terminal 300.

[0039] Fourth, the renderer 220 may constitute a transmission unit that transmits viewpoint information used to generate the specific content to the user terminal 300. The viewpoint information used to generate the specific content may be transmitted from the encoding processing unit 230 to the user terminal 300.

[0040] For example, the transmission method of the viewpoint information and the specific content may be MMTP or HTTP. When MMTP is used as the transmission method of the specific content, the viewpoint information may be stored as metadata in OMAF (Omnidirectional Media Format) defined in ISO / IEC 23090-2.

[0041] The encoding processing unit 230 encodes the specific content generated by the renderer 220. In the embodiment, the encoding processing unit 230 may be an example of a transmission unit that transmits the specific content to the user terminal 300.

[0042] Furthermore, the encoding processing unit 230 may encode the presentation time of the specific content, and may transmit an information element indicating the presentation time to the user terminal 300 together with the specific content.

[0043] Here, any compression encoding method can be used as the compression encoding method used by the encoding processing unit 230. For example, the compression encoding method may be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).

[0044] As described above, the second content included in the specific content is generated based on viewpoint information, and therefore the video included in the specific content can be treated as a 2D video that does not have freedom of viewpoint.

[0045] For example, the transmission control method used to start and end viewing of specific content may include RTSP (Real Time Streaming Protocol). The transmission method may be MMTP or HTTP. When MMTP is used as the transmission method, the specific content may be stored in OMAF defined in ISO / IEC 23090-2.

[0046] As shown in FIG. 2, the user terminal 300 includes a detection unit 310, a decoding unit 320, and a renderer 330.

[0047] The detector 310 detects the user's viewpoint position and line of sight direction. The detector 310 may include an acceleration sensor or a GPS (Global Positioning System) sensor. The detector 310 may also include a user I / F (for example, a touch sensor, keyboard, mouse, controller, etc.) that is manually input by the user. The detector 310 may transmit viewpoint information (viewpoint position and line of sight direction) to the media processing device 200. The detector 310 may output the viewpoint information (viewport) to the renderer 330.

[0048] The decoding processing unit 320 decodes the specific content received from the media processing device 200. The decoding processing unit 320 may also decode the presentation time received from the media processing device 200. The decoding processing unit 320 may output the specific content to the renderer 330, or may output the presentation time to the renderer 330.

[0049] The renderer 330 outputs the specific content decoded by the decoding processing unit 320. The renderer 330 may output the specific content based on the presentation time decoded by the decoding processing unit 320. For example, the renderer 330 may output the video content included in the specific content to a display and the audio content included in the specific content to a speaker.

[0050] Here, the renderer 330 may generate specific content with the viewpoint position and line of sight corrected based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detector 310 .

[0051] (Second content) The second content according to the embodiment will be described below. Here, the second content at t=0, t=1, and t=2 will be described. The time intervals between t=0, t=1, and t=2 are not particularly limited.

[0052] For example, as shown in Figure 3, at t = 0, the 360-degree video may be displayed without the 3D object being displayed. The 360-degree video may be considered a background video for the 3D object. At t = 1, the 3D object may be displayed in a format superimposed on the 360-degree video. Furthermore, at t = 1, the position and rotation of the 3D object superimposed on the 360-degree video may be changed.

[0053] The above-mentioned scene description includes information elements that indicate the position, rotation, and size of the 3D object for each of t=0, t=1, and t=2, allowing the 3D object to be appropriately superimposed on the 360° video.

[0054] (How to watch) A viewing method according to the embodiment will be described below, in which viewing of a specific content including a first content and a second content will be exemplified.

[0055] 4, in step S11, the user terminal 300 transmits an RTSP SETUP to the media processing device. The RTSP SETUP is a message indicating that viewing of specific content will begin.

[0056] Here, the RTSP SETUP includes the IP address of the user terminal 300, the listening port number, content identification information (content ID), etc. The RTSP SETUP may also include capability information of the user terminal 300 for viewing specific content. The capability information may include the frame rate, display resolution, etc. The display resolution may include the field of view (FoV). The capability information may also include information elements indicating the encoding method and compression method supported by the user terminal 300.

[0057] Here, a case is illustrated in which the capability information of the user terminal 300 is directly notified to the media processing device 200, but the embodiment is not limited to this. The capability information of the user terminal 300 may be notified to the transmitting device 100, and then the transmitting device 100 may notify the media processing device 200.

[0058] In step S12, the media processing device 200 transmits a response to the RTSP SETUP. Here, the response transmitted is an ACK indicating that the RTSP SETUP has been accepted.

[0059] In step S21, the user terminal 300 transmits initial viewpoint information to the media processing device 200. The initial viewpoint information may be transmitted in the MMT-SI format.

[0060] In step S22, the media processing device 200 generates initial specified content based on the initial viewpoint information (rendering process). For example, the media processing device 200 generates second content to be included in the initial specified content based on the initial viewpoint information and the scene description.

[0061] Here, media processing device 200 may generate initial specific content using a viewport that is wider than the display resolution of user terminal 300. For example, the range wider than the display resolution may be a range that is the display resolution + 20% in the horizontal direction and the display resolution + 20% in the vertical direction.

[0062] The media processing device 200 applies a compression encoding method to the initial specific content. Although not particularly limited, the compression encoding method may be HEVC or VVC.

[0063] In step S23, the media processing device 200 transmits the initial specified content corresponding to the initial viewpoint information to the user terminal 300. The media processing device 200 transmits the presentation time of the initial specified content to the user terminal 300. As described above, the presentation time (T') provided to the user terminal 300 may be determined based on the delay time (ΔT).

[0064] If a different value is used for the delay time (ΔT) for each user terminal 300, the media processing device 200 can specify this by including the transmission time of the RTSP SETUP in the RTSP SETUP described above.

[0065] The user terminal 300 outputs the specific content based on the presentation time (T'). The user terminal 300 may generate specific content with the viewpoint position and line of sight corrected based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310.

[0066] In step S31, user terminal 300 transmits viewpoint information to media processing device 200. The viewpoint information may be transmitted in the MMT-SI format. Here, user terminal 300 may transmit the viewpoint information at a predetermined interval (for example, every 500 ms), or may transmit the viewpoint information in response to a change in at least one of the viewpoint position and the line of sight direction.

[0067] In step S32, media processing device 200 generates specific content based on the viewpoint information received in step S31 (rendering process).

[0068] In step S33, media processing device 200 transmits to user terminal 300 the specific content that corresponds to the viewpoint information received in step S31.

[0069] The processing of steps S31 to S33 is the same as the processing of steps S21 to S23, except that viewpoint information received in step S31 is used instead of the initial viewpoint information. Therefore, details of the processing of steps S31 to S33 will be omitted. The processing of steps S31 to S33 may be repeated at a predetermined interval, or may be repeated every time the user's viewpoint position or line of sight direction is changed.

[0070] In step S41, the user terminal 300 transmits an RTSP TEARDOWN to the media processing device. The RTSP TEARDOWN is a message indicating that viewing of a particular content has ended.

[0071] In step S42, the media processing device 200 transmits a response to the RTSP TEARDOWN. Here, the response transmitted is an ACK indicating that the RTSP TEARDOWN has been accepted.

[0072] 4 illustrates a case where steps S11 and S12 are performed based on RTSP, but the embodiment is not limited to this. Steps S11 and S12 may be performed based on MMTP or HTTP.

[0073] Similarly, although the case where steps S41 and S42 are performed based on RTSP has been exemplified, the embodiment is not limited to this. Steps S41 and S42 may be performed based on MMTP or HTTP.

[0074] 4 illustrates an example in which steps S31 to S33 are performed based on MMTP, but the embodiment is not limited to this. Steps S31 to S33 may also be performed based on another method (for example, HTTP).

[0075] Similarly, although the case where steps S41 to S43 are executed based on MMTP has been exemplified, the embodiment is not limited to this. Steps S41 to S43 may also be executed based on another method (for example, HTTP).

[0076] (Example 1) The above-described embodiment may include the following Operation Example 1. In Operation Example 1, media processing device 200 transmits to user terminal 300 viewpoint information used in generating specific content, in association with a sequence number added to the specific content.

[0077] Specifically, as in the above-described embodiment, the media processing device 200 (renderer 220) generates second content including 360° video and 3D objects as part of specific content based on the second control information (scene description). In generating the second content, viewpoint information received from the user terminal 300 is used.

[0078] In operation example 1, the renderer 220 associates viewpoint information used in generating a specific content (here, the second content) with a sequence number. The renderer 220 transmits the viewpoint information used in generating the specific content to the user terminal 300 in association with the sequence number added to the specific content. The viewpoint information may be stored in a VP (View Port) message. The VP message may have the format of MMT-SI defined in ISO / IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc. The VP message may be transmitted for each frame. The VP message may be transmitted to the user terminal 300 as a message related to MMTP (MMT-SI).

[0079] Although not particularly limited, the VP message may have the data structure shown in Fig. 5. As shown in Fig. 5, the VP message may include message_id, version, length, fov, viewpoint_pos_x, viewpoint_pos_y, viewpoint_pos_z, viewpoint_yaw, viewpoint_pitch, viewpoint_roll, viewport_width, viewport_height, mpu_sequence_number_flag, mpu_sequence_number, etc.

[0080] The message_id is identification information indicating the VP message. The message_id may be 0x0204.

[0081] The version is information indicating the version of the MMTP protocol. The version may be 0x00.

[0082] The length is information indicating the length of the VP message.

[0083] fov is information indicating the field of view.

[0084] viewpoint_pos_x is information indicating the x coordinate of the viewpoint position. viewpoint_pos_x is an example of viewpoint information used in generating specific content.

[0085] viewpoint_pos_y is information indicating the y coordinate of the viewpoint position. viewpoint_pos_y is an example of viewpoint information used in generating specific content.

[0086] viewpoint_pos_z is information indicating the z coordinate of the viewpoint position. viewpoint_pos_z is an example of viewpoint information used in generating a specific content.

[0087] viewpoint_yaw is information indicating the yaw of the viewpoint position. viewpoint_yaw is an example of viewpoint information used in generating specific content.

[0088] viewpoint_pitch is information indicating the pitch of the viewpoint position. viewpoint_pitch is an example of viewpoint information used in generating a specific content.

[0089] viewpoint_roll is information indicating the role of the viewpoint position. viewpoint_roll is an example of viewpoint information used in generating specific content.

[0090] The viewport_width is information indicating the width of the display area (specific content).

[0091] The viewport_height is information indicating the height of the display area (specific content).

[0092] The mpu_sequence_number_flag is information indicating whether the mpu_sequence_number field exists. For example, if the mpu_sequence_number_flag is 1, the mpu_sequence_number field exists, and if the mpu_sequence_number_flag is 0, the mpu_sequence_number field does not necessarily exist.

[0093] The mpu_sequence_number is the MPU sequence number of the video corresponding to the specific content indicated by the VP message, and is an example of a sequence number added to the specific content.

[0094] Here, viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z are examples of information elements that indicate the user's viewpoint position in the three-dimensional space configured by the scene description. viewpoint_yaw, viewpoint_pitch, and viewpoint_roll are examples of information elements that indicate the user's line of sight in the three-dimensional space configured by the scene description. viewport_width and viewport_height are examples of information elements that indicate the number of pixels of the video included in specific content.

[0095] Under these conditions, the media processing device 200 and the user terminal 300 may perform the following operations.

[0096] First, the media processing device 200 (renderer 220) may determine viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z based on the viewpoint position used to generate the specific content. The renderer 220 may determine viewpoint_yaw, viewpoint_pitch, and viewpoint_roll based on the line of sight direction used to generate the specific content. The renderer 220 may determine viewport_width based on the number of pixels in the horizontal direction of the specific content, and viewport_height based on the number of pixels in the vertical direction of the specific content.

[0097] The media processing device 200 (encoding unit 230) may perform compression encoding of the specific content generated by the renderer 220 and transmit the specific content. Here, the encoding unit 230 may transmit the VP message shown in FIG. 5 to the user terminal 300. In other words, the encoding unit 230 may transmit the viewpoint information used in generating the specific content to the user terminal 300 in association with the sequence number added to the specific content.

[0098] Secondly, the user terminal 300 (decoding processing unit 320) may be configured as a receiving unit that receives viewpoint information used in generating a particular content from the media processing device 200, in association with a sequence number added to the particular content. That is, the decoding processing unit 320 may receive the VP message (MMT-SI) shown in FIG. 5 from the media processing device 200.

[0099] User terminal 300 (renderer 330) may associate the video constituting the decoded specific content with the viewpoint information included in the VP message based on the sequence number (mpu_sequence_number). Based on the difference between the viewpoint information detected by detector 310 and the viewpoint information included in the VP message, renderer 330 may identify a video in a range identified by the viewpoint information detected by detector 310 from the display area defined by the information included in the VP message (viewport_width and viewport_height). Renderer 330 may display the identified video.

[0100] (Example 2) The above-described embodiment may include the following Operation Example 2. In Operation Example 2, as shown in FIG. 6, a case is assumed in which a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed. The first user terminal 400 may be a terminal such as a head-mounted display. The first user terminal may have the same functions as the above-described user terminal 300. The second user terminal 500 may be a terminal such as a volumetric display.

[0101] Specifically, in operation example 2, as shown in FIG. 6 , the media processing device 200 (renderer 220) may constitute a receiving unit that receives content configuration and recommended viewport information from the transmitting device 100. The content configuration may be considered to include 2D video, audio, 360° video, and 3D objects. The content configuration may be considered to include MMT-SI and a scene description. The recommended viewport information may be considered to be an example of specific viewpoint information. The recommended viewport information may be information that defines (recommends) the position, direction, and angle of view at which to view video in a three-dimensional space formed by the specific content (a three-dimensional space constructed by the scene description). The recommended viewport information may include an information element that indicates at least one of an information element indicating a viewpoint position in the three-dimensional space formed by the specific content and an information element indicating a line-of-sight direction in the three-dimensional space formed by the specific content. The recommended viewport information may be considered to be viewpoint information primarily used by the second user terminal 500.

[0102] In the following, unless explicitly stated otherwise, the viewpoint information used in generating specific content may include specific viewpoint information (recommended viewport information) received from the transmitting device 100, or may include viewpoint information received from the user terminal 300.

[0103] Although not particularly limited, the recommended viewport information may be included in the scene description in the manner shown in Fig. 7. As shown in Fig. 7, the recommended viewport information may include camera_orientation, frame_number, translation, and yfov.

[0104] The camera_orientation is information that indicates the direction in which the image is viewed in the three-dimensional space constructed by the scene description. The camera_orientation can be considered to be synonymous with the line of sight.

[0105] The frame_number is information indicating the frame number of the video to which the camera_orientation, translation, and yfov are applied. The camera_orientation may be considered to be an example of specific viewpoint information.

[0106] Translation is information indicating the position from which an image is viewed in a three-dimensional space constructed by a scene description. Translation may be considered synonymous with viewpoint position. Translation may be considered an example of specific viewpoint information. For example, FIG. 7 illustrates a case where the viewpoint position is [0,0,-50] when the frame number is 0, and moves to [0,0,-75] when the frame number is 2505.

[0107] yfov is information that indicates the angle of view from which the video is viewed in the three-dimensional space constructed by the scene description.

[0108] Although not limited thereto, the camera_orientation and translation may be assigned by the content creator, or, assuming that the content is captured by a camera, the camera_orientation and translation may be assigned automatically by a GPS and a sensor provided on the camera.

[0109] Here, "translation" is an example of an information element that indicates the viewpoint position in the three-dimensional space configured by the specific content, and "camera_orientation" is an example of an information element that indicates the line of sight direction in the three-dimensional space configured by the specific content.

[0110] Under these conditions, the media processing device 200 and the user terminal 300 may perform the following operations.

[0111] First, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information). The media processing device 200 (encoding processing unit 230) may perform compression encoding of the specific content generated by the renderer 220 and transmit the specific content. Here, the encoding processing unit 230 may transmit the specific content generated based on the specific viewpoint information to the first user terminal 400, or may transmit the specific content generated based on the specific viewpoint information to the second user terminal 500.

[0112] Second, when the user terminal is a first user terminal 400 that feeds back viewpoint information, the media processing device 200 (receiving unit 210) may receive viewpoint information from the first user terminal 400. The media processing device (renderer 220) may generate specific content based on the viewpoint information received from the first user terminal 400. In such a case, the media processing device 200 (receiving unit 210) may receive a reset signal from the first user terminal 400. In response to the reset signal, the media processing device (renderer 220) may generate specific content based on the specific viewpoint information (recommended viewport information).

[0113] In such a case, the first user terminal 400 may have the same configuration as the user terminal 300. However, the first user terminal 400 (detection unit 310) may have the function of detecting a reset signal. The detection unit 310 may detect a user operation that inputs a reset signal. The detection unit 310 may also transmit the reset signal to the media processing device 200.

[0114] Third, when the user terminal is a second user terminal 500 that does not feed back viewpoint information, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information).

[0115] In such a case, the second user terminal 500 may not have the detector 310 that detects viewpoint information. The second user terminal 500 may not have the renderer 330. The second user terminal 500 may have the same configuration as the user terminal 300, except that it does not have the detector 310 or the renderer 330.

[0116] (Example 3) The above-described embodiment may include the following Operation Example 3. In Operation Example 3, media processing device 200 (for example, selection unit 260, described below) may be configured as a receiving unit that receives, from transmitting device 100, quality information about each of two or more streams whose quality varies depending on the orientation of the 3D object based on viewpoint information, as quality information about the stream related to the 3D object included in specific content.

[0117] Specifically, in operation example 3, as shown in Fig. 8, the media processing device 200 includes a selection unit 260 in addition to the components shown in Fig. 2. The selection unit 260 receives a scene description and a 3D object from the transmitting device 100. The selection unit 260 inputs a stream (3D object) selected from two or more streams to the renderer 220. The selection unit 260 may request the transmitting device 100 to transmit the selected stream. Note that in a case where the media processing device 200 transmits specific content to multiple user terminals 300, the selection unit 260 may request the transmitting device 100 to transmit the stream required by each of the multiple user terminals 300, or may request the transmitting device 100 to transmit all of the streams.

[0118] Here, the selection unit 260 receives, from the transmission device 100, quality information on each of two or more streams whose quality differs depending on the orientation of the 3D object based on the viewpoint information, as quality information on the streams related to the 3D object.

[0119] The quality information may be information indicating the relative quality of each face constituting a bounding box for a 3D object. The bounding box may be represented by a three-dimensional rectangle onto which the 3D object is projected. For example, as shown in FIG. 9, the bounding box may be defined by vertices A to H. In such a case, the faces of the bounding box include face #1 represented by vertices A, B, F, and E; face #2 represented by vertices B, C, G, and F; face #3 represented by vertices A, B, C, and D; face #4 represented by vertices E, F, G, and H; face #5 represented by vertices A, D, H, and E; and face #6 represented by vertices D, C, G, and H.

[0120] In such a case, when we imagine a 3D object being viewed in a 3D space constructed by a scene description, we assume that three sides are primarily observed, or in other words, that the remaining three sides are not observed very often.

[0121] In the third operational example, two or more streams with different qualities depending on the orientation of the 3D object based on viewpoint information are prepared as streams related to a 3D object included in specific content.

[0122] Although not particularly limited, the quality information may be included in the scene description in the manner shown in FIG. 10. FIG. 10 illustrates six streams with different orientations of 3D objects. The quality information may be expressed in the format of "quality" [#1, #2, #3, #4, #5, #6]. Note that within [ ], #1 to #6 represent the quality indexes of surfaces #1 to #6. The quality index may take on values ​​ranging from 1 to 9. A larger value of the quality index may indicate higher quality. For example, in a stream identified by "id"="1", #1, #2, and #3 have high quality ("8"), while #4, #5, and #6 have low quality ("3"). In a stream identified by "id"="2", #1, #2, and #3 have low quality ("3"), while #4, #5, and #6 have high quality ("8")

[0123] Under these conditions, the media processing device 200 may perform the following operations: The following mainly describes the selection of a stream (3D object) selected from two or more streams.

[0124] In mode 1, as shown in the upper part of Fig. 11, the media processing device 200 (selector 260) may identify the vertex closest to the user's viewpoint (for example, vertex B), and then identify the three faces that have the closest vertex (for example, face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). The selector 260 may select the stream that maximizes the sum of the quality indexes of the three identified faces (the stream identified by "id"="1" in the example shown in Fig. 10).

[0125] In mode 1, since the vertex closest to the user's viewpoint position is identified based on viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from two or more streams based on the viewpoint information and quality information.

[0126] Mode 2 may be a mode applied when a 3D object is scaled down or enlarged. For example, as shown in the middle of FIG. 11 , the media processing device 200 (selector 260) may identify the vertex closest to the user's viewpoint (e.g., vertex B), and then identify the three faces that have the closest vertex (e.g., face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). When a 3D object is scaled down, pixels of the 3D object are thinned out. Therefore, the selector 260 may select the stream with the smallest sum of the quality indexes of the three identified faces (the stream identified by "id"="2" in the example shown in FIG. 10). On the other hand, when a 3D object is scaled up, pixels of the 3D object are interpolated. Therefore, the selector 260 may select the stream with the largest sum of the quality indexes of the three identified faces (the stream identified by "id"="1" in the example shown in FIG. 10).

[0127] In mode 2, since the vertex closest to the user's viewpoint position is identified based on viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from two or more streams based on the viewpoint information and quality information.

[0128] Mode 3 may be applied when two 3D objects (3D object #1 and 3D object #2) overlap in the user's line of sight. Here, stream selection for 3D object #1 will be described. For example, as shown in the lower part of FIG. 11 , the media processing device 200 (selector 260) may identify the vertex closest to the user's viewpoint (e.g., vertex B) and then identify three faces that have the closest vertex (e.g., face #1 represented by vertices A, B, F, and E; face #2 represented by vertices B, C, G, and F; and face #3 represented by vertices A, B, C, and D). Here, 3D object #2 overlaps the line segment connecting the vertex closest to the user's viewpoint (e.g., vertex B) and the user's viewpoint, and the identified three faces are obscured by 3D object #2. Therefore, the selector 260 may select the stream for which the sum of the quality indexes of the identified three faces is the smallest (the stream identified by "id"="2" in the example shown in FIG. 10 ).

[0129] In addition, in Mode 3, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, it may be considered that the selection unit 260 selects from two or more streams the stream to be transmitted to the user terminal 300 based on the viewpoint information and quality information. Furthermore, in Mode 3, since the overlap of two 3D objects is identified based on the viewpoint information and the position information of the 3D objects, it may be considered that the selection unit 260 selects from two or more streams the stream to be transmitted to the user terminal 300 based on the viewpoint information, quality information, and position information. The position information of the 3D objects (for example, "rotation_object," "scale_object," and "translation_object" shown in FIG. 10) may be included in the scene description. The position information of the 3D objects may be considered to be "link_area" shown in FIG. 10. In other words, the selection unit 260 may receive position information of the 3D objects in three-dimensional space from the transmission device 100.

[0130] In the quality information shown in Fig. 10, the sum of the quality indexes of the six aspects is the same for each stream. However, the embodiment is not limited to this. The sum of the quality indexes of the six aspects may be different between two or more streams.

[0131] (Example 4) The above-described embodiment may include the following Operation Example 4. Here, Operation Example 4 includes the following operations in addition to Operation Example 3. In Operation Example 4, media processing device 200 (for example, selection unit 270, described below) may constitute a receiving unit that receives importance information for each of two or more objects included in specific content from transmitting device 100.

[0132] Specifically, in operation example 4, as shown in Fig. 12, the media processing device 200 has a selection unit 270 in addition to the configuration shown in Fig. 2. The selection unit 270 receives a scene description, a 3D object, and a 360° video from the transmission device 100. The selection unit 270 inputs a stream (3D object) selected from two or more streams to the renderer 220.

[0133] Here, the selection unit 270 receives importance information regarding each of the two or more objects from the transmission device 100. The objects may include a 3D object and a 360-degree video.

[0134] For example, the importance information may be information indicating the relative importance between two or more objects. For example, as shown in FIG. 13, consider a case in which object A (background), object B (person), and object C (dog) exist in a three-dimensional space constructed by a scene description. Object A (background) is an example of a 360° image, and object B (person) and object C (dog) are examples of 3D objects. In such a case, the importance information may be information indicating the relative importance between each of object A (background), object B (person), and object C (dog).

[0135] Although not particularly limited, importance information may be included in the scene description in the form shown in FIG. 14. In FIG. 14, importance information may be represented by weight. Weight may take a value ranging from 1 to 9. A larger weight value may indicate higher importance. FIG. 14 illustrates a case in which object A (background) identified by "object_id"="0" has the highest weight ("9"), object B (person) identified by "object_id"="1" has the lowest weight ("3"), and object C (dog) identified by "object_id"="2" has a weight ("8") higher than the weight of object B (person) but lower than the weight of object A (background).

[0136] Under these conditions, the media processing device 200 may perform the following operations: The following mainly describes the selection of a stream (3D object) selected from two or more streams.

[0137] First, the media processing device 200 (selector 270) selects the stream with the highest quality for the 3D object with the highest importance. The stream selection method may be the same as in operation example 3. For example, because the importance of object C (dog) is greater than the importance of object B (person), the selector 270 selects the stream for object C (dog) that maximizes the sum of the quality indexes of the three faces with the vertex closest to the user's viewpoint.

[0138] Second, media processing device 200 (selector 270) selects the lowest-quality stream for each 3D object other than the 3D object with the highest importance. Next, selector 270 replaces the lowest-quality stream with a higher-quality stream, within the scope in which a specific condition is satisfied, for each 3D object, starting with the most important. The specific condition may include a first condition that the bandwidth of the line from transmitting device 100 to media processing device 200 is equal to or less than a threshold, and a second condition that the processing load of media processing device 200 is equal to or less than a threshold. The specific condition may be defined by a combination of the first and second conditions. For example, because the importance of object B (person) is lower than the importance of object C (dog), selector 270 selects a high-quality stream for object B (person) within the scope in which the specific condition is satisfied.

[0139] As described above, it may be thought that media processing device 200 (selector 270) selects from two or more streams the stream to be transmitted to user terminal 300 based on viewpoint information, quality information, and importance information. It may also be thought that media processing device 200 (selector 270) selects from two or more streams the stream to be transmitted to user terminal 300 based on viewpoint information, quality information, placement information, and importance information.

[0140] In the fourth operational example, a case where one stream exists for 360-degree video has been exemplified. However, the embodiment is not limited to this. For 360-degree video, two or more streams of different qualities may exist.

[0141] In the fourth operational example, a case where two or more streams of different qualities exist for a 3D object depending on the orientation of the 3D object based on viewpoint information has been exemplified. However, the embodiment is not limited to this. Two or more streams of different qualities may exist for a 3D object regardless of the orientation of the 3D object.

[0142] (Example 5) The above-described embodiment may include the following Operation Example 3. In Operation Example 3, the media processing device 200 (e.g., the renderer 220) may constitute a receiving unit that receives, from the transmitting device 100, an information element that defines a range of movement of the user's viewpoint position in a three-dimensional space configured by specific content (a three-dimensional space constructed by a scene description).

[0143] First, in Operation Example 5, the information element may include an information element (hereinafter, referred to as a first information element) that restricts movement of the user's viewpoint position inside a 3D object included in specific content. For example, as shown in Fig. 15, in a case where a 3D object is placed in a three-dimensional space constructed by a scene description, movement of the viewpoint position inside the 3D object may be restricted. However, in a case where the 3D object is a building or another scene exists inside the 3D object, movement of the viewpoint position inside the 3D object may be permitted.

[0144] Second, in the fifth operational example, the information element may include an information element (hereinafter, referred to as a second information element) that restricts movement of the user's viewpoint outside the three-dimensional space. For example, as shown in FIG. 16, the three-dimensional space may be defined by a combination of a rectangular parallelepiped and a spheroid. The number of rectangular parallelepipeds that define the three-dimensional space may be two or more, and the number of spheroids that define the three-dimensional space may be two or more. However, there may be cases where movement of the user's viewpoint outside the three-dimensional space is permitted.

[0145] Although not particularly limited, the first information element may be included in the scene description in the manner shown in Fig. 17. In Fig. 17, the first information element may be represented by viewing_inside_object_flag. The viewing_inside_object_flag may be set for each 3D object. For example, when the viewing_inside_object_flag is "0", movement of the viewpoint position into the 3D object may be restricted, and when the viewing_inside_object_flag is "1", movement of the viewpoint position into the 3D object may be permitted.

[0146] Although not particularly limited, the second information element may be included in the scene description in the manner shown in Fig. 18. In Fig. 18, the second information element may include information elements (cuboid_center_x, cuboid_center_y, cuboid_center_z, cuboid_size_x, cuboid_size_y, cuboid_size_z) that define a rectangular parallelepiped that defines a three-dimensional space. The cuboid_center_x, cuboid_center_y, and cuboid_center_z are information elements that indicate the center position of the rectangular parallelepiped, and the cuboid_size_x, cuboid_size_y, and cuboid_size_z are information elements that indicate the size of the rectangular parallelepiped. The second information element may also include information elements (spheroid_center_x, spheroid_center_y, spheroid_center_z, spheroid_size_x, spheroid_size_y, spheroid_size_z) that define a spheroid that defines the three-dimensional space. spheroid_center_x, spheroid_center_y, and spheroid_center_z are information elements indicating the center position of a spheroid, and spheroid_size_x, spheroid_size_y, and spheroid_size_z are information elements indicating the size of the spheroid. Note that cuboid_enable is an information element indicating whether or not a three-dimensional space is defined by a rectangular parallelepiped, and spheroid_enable may be an information element indicating whether or not a three-dimensional space is defined by a spheroid. FIG. 18 illustrates a case in which a three-dimensional space is defined by two rectangular parallelepipeds and two spheroids.

[0147] Under these conditions, the media processing device 200 (renderer 220) may perform the following operations.

[0148] First, when the user's viewpoint position moves outside the movement range, the renderer 220 may generate specific content by using the intersection of the trajectory of the user's viewpoint position and the boundary of the movement range as the viewpoint position. In other words, the renderer 220 may fix the viewpoint position at the position (boundary position) at which the viewpoint position tries to move outside the movement range.

[0149] Second, when the user's viewpoint position moves outside the movement range, the renderer 220 may notify the user that movement of the viewpoint position is restricted. For example, the renderer 220 may display a message such as "You cannot move beyond this point."

[0150] (Action and effect) In the embodiment, media processing device 200 generates specific content based on viewpoint information and then transmits the specific content to user terminal 300. With this configuration, there is no need for user terminal 300 to generate specific content that includes second content with viewpoint flexibility; user terminal 300 can present the specific content simply by providing viewpoint information to media processing device 200. Therefore, although a delay occurs between media processing device 200 and user terminal 300, the processing load on user terminal 300 can be reduced.

[0151] In operation example 1, media processing device 200 associates the viewpoint information used in generating the specific content with the sequence number assigned to the specific content and transmits it to user terminal 300. This configuration allows user terminal 300 to grasp the viewpoint information and sequence number used in generating the specific content, and therefore allows the specific content to be displayed appropriately even in cases where the viewpoint information used when media processing device 200 generates the specific content differs from the viewpoint information used when user terminal 300 displays the specific content.

[0152] In operation example 2, media processing device 200 receives the content configuration and specific viewpoint information (recommended viewport information) from transmitting device 100. With this configuration, media processing device 200 can generate specific content based on the specific viewpoint information, and can appropriately display the specific content even in a case where a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed.

[0153] In operation example 2, the media processing device 200 generates specific content based on specific viewpoint information in response to a reset signal, even when the user terminal is the first user terminal 400. With this configuration, even in cases where the viewpoint position and line of sight direction become unknown to the first user terminal 400 in the three-dimensional space constructed by the scene description (where the user gets lost in the three-dimensional space), the reset signal allows the user to return to specific content based on specific viewpoint information.

[0154] In operation example 3, the media processing device 200 receives, from the transmitting device 100, quality information about two or more streams related to a 3D object included in specific content, the quality of which varies depending on the orientation of the 3D object based on viewpoint information. This configuration makes it possible to appropriately display a 3D object while suppressing transmission traffic, based on the new finding that the quality of each face constituting the bounding box of a 3D object does not need to be uniform.

[0155] In operation example 4, media processing device 200 receives importance information for each of two or more objects included in specific content from transmitting device 100. With this configuration, by introducing a mechanism for setting the importance of each object, such as a 360-degree video or a 3D object, it is possible to appropriately display each object included in the specific content while suppressing transmission traffic.

[0156] In operation example 5, media processing device 200 receives from transmitting device 100 information elements that define the range of movement of the user's viewpoint position in the three-dimensional space configured by the specific content. This configuration makes it possible to appropriately display specific content, including content with freedom of viewpoint, without causing any breakdown in the specific content displayed on user terminal 300.

[0157] [Change Example 1] Modification 1 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

[0158] In the first modification, a method for synchronizing the first content with the second content when the specific content includes both the first content and the second content will be described.

[0159] In the following, synchronization means that the presentation times of a first content (e.g., an MPU) and a second content (a file) are properly synchronized. Therefore, synchronization may include synchronizing the presentation times of 2D video and 3D objects, and may also include synchronizing the presentation times of audio and 3D objects. Similarly, synchronization may include synchronizing the presentation times of 2D video and 360° video, and may also include synchronizing the presentation times of audio and 360° video.

[0160] In the first method, a case will be described in which media processing device 200 synchronizes the first content and the second content based on first control information (MMT-SI). Media processing device 200 uses MMT-SI as an entry point to check whether a scene description (second content) exists, and if a scene description exists, it uses the MPU timestamp descriptor to identify the presentation time of specific content including the first content and the second content.

[0161] Specifically, as shown in FIG. 19, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 19), so that the 2D video and audio can be synchronized.

[0162] On the other hand, the presentation time of the first frame included in the scene description is specified by referencing the MPU timestamp descriptor included in the MMT-SI. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1 / 30 × n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0."

[0163] In the first method, a case has been exemplified in which the scene description does not include the presentation time of the first frame included in the scene description, but the scene description may include the presentation time of the first frame included in the scene description.

[0164] In the second method, a case will be described in which media processing device 200 synchronizes the first content and the second content based on second control information (scene description). Media processing device 200 uses the scene description as an entry point to check whether or not MMT-SI (first content) is present, and if MMT-SI is present, identifies the presentation time of specific content including the first content and the second content based on the presentation time included in the scene description.

[0165] In such a case, the scene description includes absolute time information indicating the presentation time of the second content, which may be the presentation time of the first frame included in the scene description.

[0166] For example, the absolute time information may be generated using UTC as the reference time. The reference time may be the TAI or a time provided by a GPS. The reference time may be a time provided by an NTP server or a time provided by a PTP server. Furthermore, the absolute time information may be generated based on the same reference time as the MPU timestamp descriptor.

[0167] Furthermore, the scene description includes reference information for identifying the first content. The reference information may be information for identifying an MPU that constitutes the first content. In other words, the reference information is information for treating the first content (MPU) as an object included in the scene description.

[0168] Specifically, as shown in FIG. 20, the presentation time of the first frame included in the scene description is specified by the absolute time information included in the scene description. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1 / 30×n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0."

[0169] On the other hand, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 20), so that the 2D video and audio can be synchronized. Here, because the above-mentioned reference information is included in the scene description, the media processing device 200 can check whether or not there is a first content to be presented together with the second content, based on the reference information included in the scene description.

[0170] In the second method, synchronization between 2D video and audio is achieved based on the MPU timestamp descriptor included in the MMT-SI, but in Modification 1, synchronization between 2D video and audio may also be achieved based on information elements included in the scene description (absolute time information and reference information). In such a case, at least the MPU timestamp descriptor included in the MMT-SI may be omitted. Furthermore, the MMT-SI itself may be omitted.

[0171] Note that, when the reference time of the MPU timestamp descriptor included in the MMT-SI (hereinafter referred to as the first reference time) differs from the reference time of the absolute time information included in the scene description (the second reference time), at least one of the first control information (MMT-SI) and the second control information (scene description) may include conversion information between the first reference time and the second reference time. For example, the MMT-SI may include an MPU timestamp descriptor expressed in the second reference time (e.g., a reference time other than UTC) in addition to an MPU timestamp descriptor expressed in the first reference time (e.g., UTC). The scene description may include absolute time information expressed in the first reference time (e.g., UTC) in addition to absolute time information expressed in the second reference time (e.g., a reference time other than UTC).

[0172] The MPU timestamp descriptor included in the MMT-SI may be referred to as first absolute time information, and the absolute time information included in the scene description may be referred to as second absolute time information.

[0173] [Other embodiments] Although the present invention has been described by the above disclosure, the descriptions and drawings that form part of this disclosure should not be understood as limiting the present invention. From this disclosure, various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art.

[0174] In the above disclosure, a case where the specific content includes both the first content and the second content has been exemplified, but the above disclosure is not limited to this. The specific content may include at least the second content.

[0175] Although not specifically mentioned in the above disclosure, terms related to MMT may be interpreted based on the contents specified in ISO / IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc.

[0176] In the above disclosure, an MPU timestamp descriptor is exemplified as the first absolute time information included in the MMT-SI. However, the above disclosure is not limited to this. The first absolute time information included in the MMT-SI may also be an MPU extended timestamp descriptor.

[0177] Although not specifically mentioned in the above disclosure, media processing device 200 may request part of the second content from transmitting device 100 as needed. This configuration saves bandwidth associated with transmitting the second content and prevents an increase in the processing load on media processing device 200.

[0178] In the above disclosure, MMTP is exemplified as the transmission method of the first content. However, the above disclosure is not limited to this. The transmission method of the first content may be a method conforming to ISO / IEC 23009-1 (hereinafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)). In such a case, the first control information may be MPD (Media Presentation Description). That is, in the above disclosure, MMT-SI may be read as MPD.

[0179] Although not specifically mentioned in the above disclosure, "obtain" may be read as "receive."

[0180] Although not particularly limited, operation example 2 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint flexibility, and the transmitting unit transmits specific viewpoint information used to generate specific content that includes at least the content. The receiving device includes a receiving unit that receives a content configuration with viewpoint flexibility, and the receiving unit receives the specific viewpoint information used to generate specific content that includes at least the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.

[0181] Although not particularly limited, Operation Example 3 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits quality information of a stream related to a three-dimensional object included in specific content that at least includes the content, the quality information including quality information for each of two or more streams whose quality varies depending on the orientation of the three-dimensional object. The receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives quality information of a stream related to a three-dimensional object included in specific content that at least includes the content, the quality information including quality information for each of two or more streams whose quality varies depending on the orientation of the three-dimensional object. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.

[0182] Although not particularly limited, operation example 4 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits importance information regarding each of two or more objects included in specific content that at least includes the content. The receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives importance information regarding each of two or more objects included in the specific content that at least includes the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.

[0183] Although not particularly limited, Operation Example 4 may be expressed as follows. Transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits information elements that define a range of movement of a user's viewpoint position in a three-dimensional space configured by specific content that at least includes the content. Receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives the information elements that define a range of movement of a user's viewpoint position in a three-dimensional space configured by specific content that at least includes the content. In such a case, the receiving device may be media processing device 200 or user terminal 300.

[0184] Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute the processes performed by the transmitting device 100, the media processing device 200, and the user terminal 300. The program may also be recorded on a computer-readable medium. The computer-readable medium can be used to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, DVD-ROM, or other recording medium.

[0185] Alternatively, a chip may be provided that is configured with a memory that stores programs for executing the processes performed by transmitting device 100, media processing device 200, and user terminal 300, and a processor that executes the programs stored in the memory.

[0186] The above disclosure may have the following problems and effects.

[0187] Specifically, possible user terminals for displaying content include a first user terminal that feeds back viewpoint information, such as a head-mounted display, and a second user terminal that does not feed back viewpoint information, such as a volumetric display.

[0188] After careful consideration, the inventors discovered that when considering a media device capable of generating specific content including content with freedom of viewpoint, a mechanism is needed that can appropriately display specific content even in a situation where a first user terminal and a second user terminal are mixed.

[0189] According to the above disclosure, it is possible to provide a media processing device, a transmitting device, and a receiving device that are capable of appropriately displaying specific content, including content with viewpoint flexibility.

[0190] This application claims priority from Japanese Patent Application No. 2022-019710 (filed February 10, 2022), the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0191] 10...transmission system, 100...transmitting device, 200...media processing device, 210...receiving unit, 220...renderer, 230...encoding processing unit, 260...selecting unit, 270...selecting unit, 300...user terminal, 310...detecting unit, 320...decoding processing unit, 330...renderer, 400...first user terminal, 500...second user terminal

Claims

1. a renderer that generates specific content including at least the content based on a configuration of the content having freedom of viewpoint and specific viewpoint information; an output unit that outputs the specific content generated by the renderer to a user terminal; an acquisition unit that acquires capability information of the user terminal; The renderer generates the specific content according to capability information of the user terminal; a receiving unit configured to receive viewpoint information from the first user terminal when the user terminal is a first user terminal that feeds back viewpoint information; The renderer generates the specific content based on the viewpoint information; The output unit outputs the viewpoint information used to generate the specific content, as well as the specific content generated based on the viewpoint information, to the first user terminal.

2. The media processing device according to claim 1 , wherein the specific viewpoint information includes at least one of an information element indicating a viewpoint position in a three-dimensional space formed by the specific content and an information element indicating a line-of-sight direction in the three-dimensional space.

3. a receiving unit that receives a reset signal from the first user terminal; The media processing device of claim 1 , wherein the renderer generates the specific content based on the specific viewpoint information in response to the reset signal.

4. 4. The media processing device according to claim 1, wherein the renderer generates the specific content based on the specific viewpoint information when the user terminal is a second user terminal that does not feed back viewpoint information.

5. 5. The media processing device according to claim 1, wherein the capability information of the user terminal includes information elements indicating one or more parameters selected from a frame rate of the user terminal, a display resolution of the user terminal, an encoding method of the user terminal, or a compression method of the user terminal.

Images