Information processing device, information processing method, and program
By generating and sending management files and selecting object audio data according to the display status, the problem in MPEG-DASH is solved, and the efficient transmission and rendering of object audio data is realized, reducing the storage and transmission load.
Patent Information
- Application Number
- CN201880091635.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-29
- Filing Date
- 2018-12-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2038-12-26
AI Technical Summary
In the MPEG-DASH standard, the appropriate object audio data cannot be selected according to the display status, resulting in low transmission and rendering efficiency. The prior art requires the transmission of unnecessary object audio data in different display statuses, which increases the storage and transmission load.
By generating and sending management files, the object selection information containing the object audio data is selected, the appropriate object audio data is selected according to the display status of the image, and the dynamic selection and file processing of object audio data is realized in the server and client to ensure that only Transfer and render appropriate audio data when needed.
The selection of object audio data according to the display status is realized, reducing transmission and storage load, improving the transmission efficiency and user experience of audio data, and reducing the processing requirements for clients.
Smart Images

Figure CN111903135B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art
[0002] In recent years, MPEG-DASH (Moving Picture Experts Group phase-Dynamic Adaptive Streaming over HTTP) has become increasingly popular as a basic technology for Internet streaming services (see, for example, Non-Patent Document 1).
[0003] In the delivery of audio content performed using MPEG-DASH, adaptive streaming is achieved by a delivery server that prepares audio data for each object (the data will be referred to as "object audio data") and a client that requests optimal object audio data based on the conditions of the transmission path, etc.
[0004] Furthermore, as a coding technology for realizing reproduction with a higher sense of reality beyond conventional 5.1 channel surround reproduction and transmission of multiple pieces of object audio data, MPEG-H 3D Audio is known (see, for example, Non-Patent Document 2).
[0005] Reference List
[0006] Non-patent literature
[0007] Non-patent document 1: ISO / IEC 23009-1:2014 Information technology - Dynamic adaptive streaming over HTTP (DASH)
[0008] Non-patent document 2: ISO / IEC 23008-3:2015Information technology-Highefficiency coding and media delivery in heterogeneous environments Summary of the invention
[0009] Technical issues
[0010] However, in the standard of MPEG-DASH described in Non-Patent Document 1 as described above, it is not possible to select target audio data to be reproduced for each display state.
[0011] Therefore, the present disclosure proposes a novel and improved information processing device, information processing method, and program, which are capable of selecting target audio data according to a display state.
[0012] Solution to the problem
[0013] According to the present disclosure, there is provided an information processing apparatus including: a generating unit configured to generate a management file storing information related to object selection information for selecting object audio data according to a display state of an image; and a transmitting unit configured to transmit the management file.
[0014] In addition, according to the present disclosure, there is provided an information processing method to be performed by an information processing device, the information processing method including: generating a management file storing information related to object selection information for selecting object audio data according to a display state of an image; and sending the management file.
[0015] Furthermore, according to the present disclosure, there is provided a program that causes a computer to realize the following functions: a function of generating a management file storing information related to object selection information for selecting object audio data according to a display state of an image; and a function of transmitting the management file.
[0016] Advantageous Effects of the Invention
[0017] As described above, according to the present disclosure, target audio data can be selected according to the display state.
[0018] Note that the above effects are not necessarily limitative, and any effects described in the present specification may be obtained together with or instead of the above effects or other effects that may be detected from the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is an explanatory diagram for explaining the background of the present disclosure.
[0020] Figure 2 This is an explanatory diagram for explaining the background of the present disclosure.
[0021] Figure 3 This is an explanatory diagram for explaining the relationship between the display state of an image and target audio data.
[0022] Figure 4A is an explanatory diagram for explaining the transmission of audio data according to the first comparative method.
[0023] Figure 4B is an explanatory diagram for explaining the transmission of audio data according to the second comparison method.
[0024] Figure 5: is an explanatory diagram for explaining the overview of a proposed method according to an embodiment of the present disclosure.
[0025] Figure 6 : is an explanatory diagram for explaining an example in which a combination of target audio data to be utilized changes according to time even if a display state does not change.
[0026] Figure 7 is a block diagram showing a system configuration example of an information processing system according to an embodiment.
[0027] Figure 8 is a block diagram showing a functional configuration example of the server 100 according to the embodiment.
[0028] Fig. 9 is a block diagram showing a functional configuration example of the client 200 according to the embodiment.
[0029] Fig. 10A 2 is an explanatory diagram for explaining target audio data to be utilized in the case of displaying a decoded image as it is on the display unit 260 .
[0030] Fig. 10B 2 is an explanatory diagram for explaining target audio data to be utilized in the case of displaying a cutout image obtained by cutting out a decoded image on the display unit 260 .
[0031] Fig.11 This is an explanatory diagram for explaining the difference in target audio data to be used due to the difference in viewing angles during viewing.
[0032] Fig.12 is a flowchart illustrating a method in which the client 200 selects and acquires object audio data to be utilized based on object selection information.
[0033] Fig.13 This is an illustration for explaining the angle between objects.
[0034] Fig.14 is a flowchart illustrating a method in which the client 200 selects and acquires object audio data to be utilized based on object selection information.
[0035] Fig.15 It is a flowchart showing the process of filing.
[0036] Fig.16 This is an explanatory diagram for explaining a specific example of documentation.
[0037] Fig.17 This is an explanatory diagram for explaining a specific example of documentation.
[0038] Fig.18This is an explanatory diagram for explaining a specific example of documentation.
[0039] Fig.19 It is a table showing the values to be transmitted by signal and the detailed description of each value.
[0040] Fig. 20 is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 .
[0041] Fig.21 It is shown Fig. 20 A table of the various pieces of information in the MPD file shown.
[0042] Fig. 22 is a flowchart showing an example of a method for selecting an AdaptationSet acquired by the client 200 .
[0043] Fig.23 is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 .
[0044] Fig.24 It is a table showing the values to be transmitted by signal and the detailed description of each value.
[0045] Fig.25 is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 .
[0046] Fig.26 It is shown Fig.25 A table of the various pieces of information in the MPD file shown.
[0047] Fig. 27 is a flowchart illustrating an example of a method for selecting an AdaptationSet acquired by the client 200 .
[0048] Fig.28 is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 .
[0049] Fig.29 This is an explanatory diagram for explaining the structure of a metadata file.
[0050] Fig.30 This is an explanatory diagram for explaining the structure of a metadata file.
[0051] Fig.31 It is shown Fig.29 and Fig.30 A view showing the syntax of the metadata file.
[0052] Fig.32 is a view showing an example of an extended metadata file.
[0053] Fig.33 is a view showing the syntax of an extended metadata file.
[0054] Fig.34 is a view showing an example of an object selection metadata file.
[0055] Fig.35 is a view showing the syntax of an object selection metadata file.
[0056] Fig.36 is a view showing an example of an objectSelectionMetadataConfigurationBox (object selection metadata configuration box).
[0057] Fig.37 It is shown Fig.36 A table showing the value of objectSelectionMetadataConfigurationBox and a detailed description of each value.
[0058] Fig.38 is a view showing an example of objectSelectionMetadata (object selection metadata).
[0059] Fig.39 It is shown Fig.38 A table showing the value of objectSelectionMetadata and a detailed description of each value.
[0060] Fig.40 is a view showing a modified example of objectSelectionMetadata.
[0061] Fig.41 It is shown Fig.40 A table with detailed descriptions of each value shown.
[0062] Fig.42 is a view showing an example of an objectSelectionMetadataConfigurationBox.
[0063] Fig.43 It is shown Fig.42 A table showing the value of objectSelectionMetadataConfigurationBox and a detailed description of each value.
[0064] Fig.44 is a view showing an example of objectSelectionMetadata.
[0065] Fig.45 It is shown Fig.44 A table showing the value of objectSelectionMetadata and a detailed description of each value.
[0066] Fig.46 is a view showing an example of objectSelectionMetadata.
[0067] Fig.47 It is shown Fig.46 A table showing the value of objectSelectionMetadata and a detailed description of each value.
[0068] Fig.48 is a view showing a modified example of objectSelectionMetadata.
[0069] Fig.49 It is shown Fig.48 A table with detailed descriptions of each value shown.
[0070] Fig.50 is a view showing an example of an objectSelectionMetadataConfigurationBox.
[0071] Fig.51 It is shown Fig.50 A table showing the value of objectSelectionMetadataConfigurationBox and a detailed description of each value.
[0072] Fig.52 is a view showing an example of an MPD file to be generated.
[0073] Fig.53 is a view showing an example of an MPD file to be generated.
[0074] Fig.54 is a view showing another example of an MPD file to be generated.
[0075] Fig.55 is a flow chart illustrating an example of a method for selecting a file to be acquired by a client.
[0076] Fig.56This is an explanatory diagram for explaining an application example.
[0077] Fig.57 This is an explanatory diagram for explaining an application example.
[0078] Fig.58 This is an explanatory diagram for explaining an application example.
[0079] Fig.59 is a view showing an example of an MPD file to be generated in a case where target audio data to be utilized differs according to a viewing position.
[0080] Fig.60 is an explanatory diagram showing an example of a hardware configuration.
[0081] Fig.61 is a view showing the structure of MHAMultiStreamBox.
[0082] Fig.62 is a view showing the structure of 3da_meta_data().
[0083] Fig.63 is a view showing the structure of DSE.
[0084] Fig.64 is a view showing the structure of 3da_ancillary_data to be stored in data_stream_byte in DSE. DETAILED DESCRIPTION
[0085] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the accompanying drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.
[0086] In addition, in the present specification and the accompanying drawings, in some cases, multiple components having substantially the same functional configuration will be distinguished by giving the same reference numerals followed by different letters. However, in the case where each of the multiple components having substantially the same functional configuration does not need to be particularly distinguished, only the same reference numerals will be attached.
[0087] The description will be given in the following order.
[0088] <<1. Introduction>>
[0089] <<2. Configuration Example>>
[0090] <<3. Example of object selection information (Example 1)>>
[0091] <<4. Example of considering documenting display status (Example 2)>>
[0092] <<5. Example of signal transmission in a case where a combination of object audio data does not change over time (Example 3)>>
[0093] <<6. Example of signal transmission in a case where combination of object audio data changes over time (Example 4)>>
[0094] <<7. Application Examples>>
[0095] <<8. Hardware Configuration Example>>
[0096] <<9. Conclusion>>
[0097] <<1. Introduction>>
[0098] <1-1. Background>
[0099] First, the background of the present disclosure will be described.
[0100] MPEG-H 3D Audio is a standard that can process object audio data, which is audio data generated for each object. Audio content consists of multiple pieces of object audio data, which are waveform data of the sound source, and object metadata including information about the position of the object, propagation of the sound, various types of effects, etc.
[0101] Figure 1 is an illustration diagram for explaining the background of the present disclosure. Figure 1 As shown, the object metadata and multiple pieces of object audio data (in Figure 1 , it is shown that object audio data 1 to object audio data n) are provided to a client by a server, etc. When an object renderer of the client receives object metadata and object audio data, the object renderer performs rendering based on reproduction environment information (e.g., the position and number of speakers, etc.), and reproduces audio content by providing waveform data to the reproduction environment such as speakers.
[0102] In the case of providing audio content, the object audio data is stored in an audio file and sent to the client. Figure 2 A specific example is described. Figure 2 is an illustration diagram for explaining the background of the present disclosure. Figure 2 In the case where there are object audio data 1 to object audio data 3 and object metadata corresponding to these data as shown in D10 in FIG, there are roughly three possible modes of storage in the audio file as shown in filing examples F11 to F13.
[0103] More specifically, as shown in the example of file F11, all data can be stored in one audio file, or as shown in the example of file F12, one or two or more pieces of object audio data and object metadata corresponding to these pieces of data can be stored in audio files, respectively. In addition, as shown in the example of file F13, one piece of object audio data and object metadata corresponding to the data can be stored in one audio file.
[0104] Here, we will refer to Figure 3 Describes the relationship between the display status of the image and the object audio data. Figure 3 This is an explanatory diagram for explaining the relationship between the display state of an image and target audio data.
[0105] Figure 3 An example is shown in which an image in which two singers H2 and H3 are singing is displayed in two display states V11 and V12. The display state V11 and the display state V12 can be switched to each other by zooming in and out, and if the center part is zoomed in in the display state V11, the image changes to the display state V12, and if the image is zoomed out in the display state V12, the image changes to the display state V11. Note that zooming in and out can be performed by, for example, image cropping processing performed by the client.
[0106] In the case of display state V11, because the two singers H2 and H3 are compactly displayed in the center, even if one of the singers is singing or both singers are singing, it can be considered that it is sufficient as long as the sound comes from one direction in the center. Meanwhile, in the case of display state V12, because the two singers appear to be separated from each other from the user's point of view, and the user can clearly distinguish the positions of the respective singers, it is expected that the sound comes from the left direction when the left singer H2 is singing, and the sound comes from the right direction when the right singer H3 is singing. This state is called a state in which sound localization is known.
[0107] In MPEG-DASH which is a standard for performing a streaming service, transmission of target audio data according to a display state as described above is not considered, and the target audio data is transmitted regardless of the display state.
[0108] For example, when using MPEG-DASH to transmit Figure 3 In the case of the contents of the image and audio data shown, two methods (which will be referred to as a first comparison method and a second comparison method, respectively) which will be described below with reference to FIG. 4 may be employed.
[0109] Figure 4A1 is an explanatory diagram for explaining the transmission of audio data according to the first comparison method. The first comparison method is a method of transmitting a piece of object audio data AD1 according to the display state V11. Figure 4A As shown, according to the first comparison method, since the sound in the case of display state V12 and in the case of display state V11 also comes from only one direction of the center, it is not possible to compare the sound in the case of display state V12 and in the case of display state V11. Figure 3 As described, the sound is heard from the direction according to the position of each singer.
[0110] Figure 4B 1 is an explanatory diagram for explaining the transmission of audio data according to the second comparative method. The second conventional method is a method of transmitting two pieces of object audio data AD2 and AD3 according to the display state V12. Figure 4B As shown, according to the second comparison method, although the sound can be heard from the direction according to the position of each singer in the display state V12, in the display state V11, two pieces of object audio data are rendered, and the resultant sound comes from one direction. That is, with the second comparison method, in the case of the display state V11, since two pieces of object audio data are still required although it is sufficient for the sound to come from only one direction, an additional transmission band and rendering processing are required compared to the first comparison method.
[0111] <1-2. Overview>
[0112] Therefore, a method for selecting and acquiring object audio data to be utilized according to a display state (hereinafter referred to as the "proposed method") is adopted in the embodiments of the present disclosure to be described below. The proposed method can be implemented by, for example, the following operations: the server pre-stores all object audio data required for each display state, and the client selects and acquires the object audio data according to the display state.
[0113] Figure 5 1 is an explanatory diagram for explaining an overview of the proposed method according to an embodiment of the present disclosure. In the proposed method, one piece of object audio data AD1 is transmitted in display state V11, and two pieces of object audio data AD2 and AD3 are transmitted in display state V12. According to the proposed method, it becomes possible to transmit the object audio data without causing an additional transmission band and rendering processing, so that the sound comes from a more appropriate position.
[0114] Incidentally, as described above, in the existing MPEG-DASH, the transmission of the object audio data according to the display state is not considered. Therefore, in order to realize the transmission of the object audio data using the proposed method as described above, in the present disclosure, the following four mechanisms are provided.
[0115] First, information for selecting target audio data according to the display state of an image and a mechanism for selecting target audio data by utilizing the information are provided. Figure 5 In the example shown, meta information (hereinafter, also referred to as object selection information) for selecting the object audio data AD1 in the case of display state V11 and selecting the object audio data AD2 and AD3 in the case of display state V12 is newly required. Such object selection information and an example of selecting object audio data using the object selection information will be described in <<3. Example of object selection information (Example 1)>>.
[0116] Second, a mechanism is provided for documenting the object audio data taking into account the display state. Figure 2 As described above, in the existing MPEG-DASH, object audio data is stored in an audio file, and the client selects and obtains the file.
[0117] For example, in Figure 2 In the case where all object audio data to be used for each display state are filed so as to be stored in one audio file as in the shown filing example F11, there is provided an advantage that only one audio file needs to be selected and acquired at all times according to the display state. However, it is necessary to prepare the same number of files as the number of display states, and in the case where there are a plurality of display states utilizing common object audio data, the server needs to store a plurality of audio files including the same object audio data, which may result in a waste of storage capacity.
[0118] At the same time, in Figure 2 In the case where one object audio data is filed so as to be stored in one audio file as in the filing example F13 shown, there is an advantage that the server only needs to store audio files corresponding to the required object audio data. However, since when the client obtains the file, it is necessary to obtain the number of audio files corresponding to the number of pieces of object audio data according to the display state, there is a possibility that the number of HTTP connections may increase and the processing load of the server may increase.
[0119] Therefore, a mechanism is desired that can support various display states and file the object audio data while suppressing the amount of data to be stored in the server and the number of files to be acquired by the client. An example of such a mechanism for filing the object audio data in consideration of the display state will be described in <<4. Example of filing in consideration of display state (Example 2)>>.
[0120] Third, in the case where the combination of object audio data to be utilized does not change over time, a mechanism for transmitting object selection information using a signal is provided. In the case where the combination of object audio data to be utilized does not change over time, the same object selection information can be commonly utilized at all times. However, since such a mechanism for transmitting object selection information using a signal is not prepared in the existing MPEG-DASH, a new signal transmission mechanism is required. An example of a mechanism for transmitting object selection information using a signal in the case where the combination of object audio data to be utilized does not change over time will be described in <<5. Example of signal transmission in the case where the combination of object audio data does not change over time (Example 3)>>.
[0121] Fourth, in the case where the object audio data to be utilized changes over time, a mechanism for signaling object selection information is provided. There may be a case where the above object selection information changes over time. For example, there is a case where the object audio data to be utilized changes according to time even in the same display state. Figure 6 Describe such a situation.
[0122] Figure 6 : is an explanatory diagram for explaining an example in which a combination of target audio data to be utilized changes according to time even if the display state does not change. Figure 6 In the example shown in , although the zoom does not change from time t1 to time t2, and the display state V21 at time t1 is the same as the display state V22 at time t2, the two singers H1 and H2 are closer to the front side from time t1 to time t2. Note that in this specification, in the case where an element such as a viewing angle that can determine the display state does not change, for example, the state is expressed as the same display state.
[0123] Since the two singers H1 and H2 seem to be together in distance at time t1, it is desirable to utilize one piece of object audio data AD1. Furthermore, at time t2, since the positions of the respective singers can be clearly distinguished as a result of the two singers H1 and H2 being closer, it is desirable to utilize two pieces of object audio data AD2 and AD3. In this way, in the case where the sound source in the image moves at each moment, the combination of the object audio data to be utilized can change over time. Furthermore, in the case where the sound source in the image changes at each moment, the combination of the object audio data to be utilized can also change over time.
[0124] In the case where the combination of object audio data to be utilized changes over time in this way, it is desirable to transmit object selection information that is different at each moment by signal. In the existing MPEG-DASH, there is no mechanism for storing object selection information that is different at each moment in a file and for transmitting the file by signal. An example of a mechanism for storing object selection information in a file at each moment and transmitting the file by signal in the case where the combination of object audio data to be utilized changes over time will be described in <<6. Example of signal transmission in the case where the combination of object audio data changes over time (Example 4)>>.
[0125] The background of the present disclosure and the overview of the embodiments of the present disclosure have been described above. The embodiments of the present disclosure will be described in more detail below.
[0126] <<2. Configuration Example>>
[0127] <2-1. System Configuration Example>
[0128] First, refer to Figure 7 A system configuration example of the information processing system according to the present embodiment is described. Figure 7 is a block diagram showing a system configuration example of an information processing system according to the present embodiment.
[0129] Reference Figure 7 , the information processing system according to the present embodiment includes a server 100 and a client 200. In addition, the server 100 and the client 200 are connected to each other through a communication network 300.
[0130] The server 100 is an information processing device (transmitting device) that streams (transmits) object audio data to be used for audio content to the client 200 based on MPEG-DASH. More specifically, the server 100 acquires audio data to be used for audio content for each object, and generates streaming data by encoding the data in units of objects. In addition, the server 100 generates an audio file by filing the streaming data for each time unit (which is called a "segment") from several seconds to about 10 seconds or for all content.
[0131] Note that an object is a sound source, and audio data of each object (object audio data) is acquired using, for example, a microphone attached to each object, etc. The object may be a subject such as a fixed microphone stand, or may be a dynamic subject such as a person.
[0132] Furthermore, in this embodiment, one sound source may be one object, or multiple sound sources may be one object. Furthermore, multiple objects may exist to correspond to a given sound source. For example, in Figure 5In the illustrated example, the object corresponding to the singer H2 as the sound source differs depending on the display state, and the singing voice of the singer H2 may be included in both the object audio data AD1 and the object audio data AD2 .
[0133] Furthermore, the server 100 encodes object metadata including location information of each object, etc. The server 100 generates a metadata file by filing the encoded data of the object metadata in units of segments.
[0134] In addition, the server 100 generates a management file for managing the audio file. In the present embodiment, the management file may be a Media Presentation Description (MPD) file defined in the ISO / IEC 23009-1 standard.
[0135] In addition, the server 100 sends the above-mentioned audio files, metadata files, MPD files, etc. to the client 200 in response to a request from the client 200.
[0136] The client 200 is an information processing device (receiving device) that reproduces audio content. More specifically, the client 200 obtains an MPD file from the server 100, and based on the MPD file, obtains a metadata file and an audio file from the server 100. Then, the client 200 decodes the audio file obtained from the server 100, and realizes the reproduction of the audio content by synthesizing and outputting the audio file.
[0137] The system configuration example of the information processing system according to the present embodiment has been described above. Figure 7 The above configuration described is only an example, and the configuration of the information processing system according to the present embodiment is not limited to such an example. For example, part of the functions of the server 100 may be provided at the client 200 or other external devices. For example, software that provides part of the functions of the server 100 (e.g., a WEB application using a predetermined application programming interface (API) or the like) may be executed on the client 200. In addition, conversely, part of the functions of the client 200 may be provided at the server 100 or other external devices. The configuration of the information processing system according to the present embodiment may be flexibly changed according to specifications and operations.
[0138] <2-2. Example of server functional configuration>
[0139] The system configuration example of the information processing system according to the present embodiment has been described above. Figure 8 A functional configuration example of the server 100 is described. Figure 8 1 is a block diagram showing a functional configuration example of the server 100 according to the present embodiment. Figure 8As shown, the server 100 includes: a generating unit 110 , a control unit 120 , a communication unit 130 and a storage unit 140 .
[0140] The generation unit 110 is a functional component that performs processing related to the provision of audio content. Figure 8 As shown, the generating unit 110 includes: a data acquiring unit 111, an encoding processing unit 112, a segment file generating unit 113 and an MPD file generating unit 114.
[0141] The data acquisition unit 111 acquires audio data of each object (in other words, acquires object audio data) and an image. The data acquisition unit 111 may acquire the image and object audio data from the inside of the server 100, or may acquire the image and object audio data from an external device connected to the server 100. The data acquisition unit 111 provides the acquired image and object audio data to the encoding processing unit 112.
[0142] The encoding processing unit 112 generates image stream data and audio stream data by encoding the image and the object audio data of each object provided from the data acquisition unit 111. In addition, the encoding processing unit 112 encodes object metadata including object position information of each object and the like input from the outside. The encoding processing unit 112 provides the encoded data of the image stream data, the audio stream data of each object, and the object metadata to the segment file generation unit 113.
[0143] The segment file generation unit 113 is a functional component that generates a segment file, which is data in which data can be transmitted as a unit of content. For example, the segment file generation unit 113 generates an audio file by filing the audio stream data of each object provided from the encoding processing unit 112 in units of segments. There are various aspects of the generation of audio files (also referred to as "filing"). For example, the segment file generation unit 113 generates an audio file in the following manner: based on the combination of object audio data according to the display state, one or two or more object audio data are stored in one audio file so that the same object audio data is not included in multiple audio files. The details of generating an audio file based on the combination of object audio data according to the display state will be described later.
[0144] Furthermore, the segment file generating unit 113 may also generate a metadata file storing only the object metadata by filing, in segments, the encoded data of the object metadata supplied from the encoding processing unit 112. The case of generating such a metadata file will be described later.
[0145] Furthermore, the segment file generation unit generates an image file by filing the image stream data supplied from the encoding processing unit 112 in units of segments.
[0146] The MPD file generation unit 114 is a functional component for generating an MPD file. In the present embodiment, the MPD file generation unit 114 generates an MPD file including metadata file access information, which is used to access the metadata file including object selection information. The MPD file generated by the MPD file generation unit 114 will be described in detail later.
[0147] The control unit 120 is a functional component that comprehensively controls the overall processing to be performed by the server 100. For example, the control unit 120 can control the start and stop of each component based on request information or the like received from the client 200 via the communication unit 130. Note that the control content of the control unit 120 is not particularly limited. For example, the control unit 120 can control processing generally performed in a general-purpose computer, a PC, a tablet PC, etc.
[0148] The communication unit 130 performs various types of communications with the client 200. For example, the communication unit 130 receives request information from the client 200. In addition, the communication unit 130 also functions as a sending unit, and sends an MPD file, a metadata file, an audio file, an image file, etc. to the client 200 in response to the request information. Note that the communication contents of the communication unit 130 are not limited to these.
[0149] The storage unit 140 is a functional component that stores various types of information. For example, the storage unit 140 stores audio files, image files, metadata files, MPD files, etc., or stores programs, parameters, etc. to be used by various functional components of the server 100. Note that the information stored in the storage unit 140 is not limited to these.
[0150] The functional configuration example of the server 100 has been described above. Figure 8 The functional components described are merely examples, and the functional configuration of the server 100 is not limited to the examples. For example, the server 100 does not necessarily have to include Figure 8 All functional components are shown. In addition, the functional configuration of the server 100 can be flexibly changed according to specifications and operations.
[0151] <2-3. Client Function Configuration Example>
[0152] The functional configuration example of the server 100 has been described above. Fig. 9 A functional configuration example of the client 200 is described. Fig. 92 is a block diagram showing a functional configuration example of the client 200 according to the present embodiment. Fig. 9 As shown, the client 200 includes: a processing unit 210 , a control unit 220 , a communication unit 230 , a storage unit 240 , a sensor unit 250 , a display unit 260 and a speaker 270 .
[0153] The processing unit 210 is a functional component that performs processing related to content reproduction. Fig. 9 As shown, the processing unit 210 includes: an MPD file acquisition unit 211, an MPD file processing unit 212, a segment file selection unit 213, a segment file acquisition unit 214, a decoding processing unit 215 and a synthesis processing unit 216.
[0154] The MPD file acquisition unit 211 is a functional component that acquires an MPD file from the server 100 before content reproduction. More specifically, the MPD file acquisition unit 211 acquires an MPD file from the server 100 by generating request information for the MPD file based on a user operation or the like, and providing the request information to the server 100 via the communication unit 230. The MPD file acquisition unit 211 provides the acquired MPD file to the MPD file processing unit 212.
[0155] The MPD file processing unit 212 is a functional unit that performs processing related to the MPD file provided from the MPD file acquisition unit 211. More specifically, the MPD file processing unit 212 identifies information (e.g., URL, etc.) required to acquire an audio file, a metadata file, etc. based on the analysis of the MPD file. In addition, the MPD file processing unit 212 according to the present embodiment acquires object selection information based on the analysis of the MPD file. The MPD file processing unit 212 provides these types of information to the segment file selection unit 213.
[0156] The segment file selection unit 213 is a functional component that selects a segment file to be acquired. More specifically, the segment file selection unit 213 selects an audio file or a metadata file to be acquired based on the above-mentioned various types of information provided from the MPD file processing unit 212. For example, the segment file selection unit 213 according to the present embodiment selects object audio data to be utilized based on object selection information, and selects an audio file including the selected object audio data.
[0157] The segment file acquisition unit 214 is a functional component that acquires segment files. More specifically, the segment file acquisition unit 214 acquires image files, audio files, or metadata files from the server 100 by generating request information for these files based on various types of information provided from the segment file selection unit 213, and sending the request information to the server 100 via the communication unit 230. The segment file acquisition unit 214 provides the acquired files to the decoding processing unit 215.
[0158] The decoding processing unit 215 is a functional component that decodes data included in the image file, audio file, or metadata file provided from the segment file acquisition unit 214. The decoding processing unit 215 provides an image obtained by the decoding process (hereinafter, also referred to as a "decoded image") to the control unit 220. In addition, the decoding processing unit 215 provides the object audio data and the like obtained by the decoding process to the synthesis processing unit 216.
[0159] The synthesis processing unit 216 is a functional section that synthesizes the plurality of pieces of target audio data supplied from the decoding processing unit 215 and outputs the synthesized data. The synthesis processing unit 216 supplies the synthesized data to the control unit 220.
[0160] The control unit 220 is a functional component that comprehensively controls the overall processing to be performed by the client 200. For example, the control unit 220 controls the display of the display unit 260. The control unit 220 may cause the decoded image provided from the decoding processing unit 215 to be displayed at the display unit 260 as it is, or may perform processing on the decoded image provided from the decoding processing unit 215 and cause the decoded image subjected to the processing to be displayed at the display unit 260. For example, the control unit 220 may perform processing (cutting processing) of cutting a part of an area from the decoded image provided from the decoding processing unit 215 based on the sensor data provided from the sensor unit 250. Then, the control unit 220 may cause the image cut by the cutting processing (hereinafter, also referred to as "cutting image") to be displayed at the display unit 260. Note that the cutting processing of the decoded image based on the sensor data may be similar to the cutting processing to be performed in the reproduction of, for example, a so-called 360-degree image. In addition, the control unit 220 provides audio content to the user by controlling the speaker 270 and outputting the synthesized data provided from the synthesizing processing unit 216. In addition, the control unit 220 can control various types of processing based on input performed by the user using an input unit (not shown) such as a mouse and a keyboard. Note that the control content of the control unit 220 is not particularly limited. For example, the control unit 220 can control processing generally performed at a general-purpose computer, a PC, a tablet PC, etc.
[0161] The communication unit 230 performs various types of communications with the server 100. For example, the communication unit 230 transmits the request information provided from the processing unit 210 to the server 100. In addition, the communication unit 230 also functions as a receiving unit, and receives an MPD file, a metadata file, an audio file, an image file, etc. from the server 100 in response to the request information. Note that the communication contents of the communication unit 230 are not limited to these.
[0162] The storage unit 240 is a functional component that stores various types of information. For example, the storage unit 140 stores image files, audio files, object metadata, metadata files, MPD files, etc. acquired from the server 100, or stores programs, parameters, etc. to be used by various functional components of the client 200. Note that the information stored in the storage unit 240 is not limited to these.
[0163] The sensor unit 250 acquires sensor data about the user through sensing. For example, the sensor unit 250 may include sensors such as an acceleration sensor and an angular velocity sensor, and may acquire sensor data related to the user's posture and position. The sensor unit 250 provides the acquired sensor data to the control unit 220.
[0164] The display unit 260 displays an image according to the control of the control unit 220. Note that the image displayed on the display unit 260 may be an intact decoded image supplied from the decoding processing unit 215, or may be a cutout image cutout by the cutout processing of the control unit 220, as described above.
[0165] The speaker 270 performs audio output according to the control of the control unit 220 .
[0166] The functional configuration example of the client 200 has been described above. Fig. 9 The functional components described are merely examples, and the functional configuration of the client 200 is not limited to the examples. For example, the client 200 does not necessarily have to include Fig. 9 In addition, the functional configuration of the client 200 can be flexibly changed according to the specifications and operations. For example, Fig. 9 The illustrated functional components of the sensor unit 250 , the display unit 260 , and the speaker 270 may be provided at other devices connected to the client 200 .
[0167] <<3. Example of object selection information (Example 1)>>
[0168] The configuration example of the present embodiment has been described above. Subsequently, two examples (Example 1-1 and Example 1-2) related to object selection information for selecting object audio data to be utilized according to a display state and selection of object audio data using the object selection information in the present embodiment will be described.
[0169] <3-1. Example of object selection information using perspective (Example 1-1)>
[0170] First, as Example 1-1, an example of object selection information using a viewing angle will be described. As described above, the control unit 220 of the client 200 may cause the decoded image provided from the processing unit 210 to remain as it is at the display unit 260, or may cause a cut image cut by a cut process to be displayed at the display unit 260. Here, in the case where the decoded image is displayed as it is and in the case where the cut image is displayed, the display state may be different, and the object audio data to be acquired by the client 200 may be different. This difference in the object audio data to be acquired by the cut process will be described with reference to FIG. 10. Note that although the horizontal viewing angle will be mainly described in the following description, the description of the vertical viewing angle will be similar.
[0171] Fig. 10A 2 is an explanatory diagram for explaining target audio data to be utilized in the case of displaying a decoded image as it is on the display unit 260 . Fig. 10A The decoded image D31 shown is based on the viewing angle θ a The image is imaged, and there are two singers H1 and H2 in the decoded image D31.
[0172] Such a decoded image D31 is displayed as it is and at the same viewing angle θ as that at the time of imaging. a When viewed, the two singers H2 and H3 are compactly displayed in the center, as in Fig. 10A Therefore, in the display state V32, it is considered that it is sufficient as long as the sound comes from one direction. In this case, the client 200 only needs to select and obtain one object audio data.
[0173] Notice, Fig. 10A The example shown can be interpreted as the same viewing angle θ as that at the time of imaging. a An example in which the entire decoded image is cut out as a cut out image and displayed.
[0174] Fig. 10B 2 is an explanatory diagram for explaining target audio data to be utilized in the case where a cutout image obtained by cutting out a decoded image is displayed on the display unit 260. Fig. 10A In a similar manner to the decoded image D31 shown, Fig. 10B The decoded image D33 shown is based on the viewing angle θ a The image is imaged, and there are two singers H1 and H2 in the decoded image D33.
[0175] Here, at a viewing angle θ b The cutout image D331 cutout from such a decoded image D33 is displayed and is displayed at a viewing angle θ a In the case of viewing, the two singers appear to be separated from each other to the user, as in Fig. 10B As shown in the display state V34, the user can clearly distinguish the positions of each singer. Therefore, in the display state V34, the desired sound comes from two directions respectively. In this case, the client 200 needs to select and obtain two pieces of object audio data.
[0176] As described above, in the case where the decoded image is displayed as it is and in the case where the cut-out image is displayed, the display state may be different and the object audio data to be utilized may be different. Furthermore, even in the case where the cut-out image is displayed, the display state and the object audio data to be utilized may be different depending on the viewing angle (for example, in Fig. 10B In the example shown, θ b ) and the direction (position) of shearing the sheared image.
[0177] Furthermore, even in the case where the displayed cutout image is the same, the display state and the object audio data to be utilized may be different depending on the viewing angle at the time of viewing (hereinafter, also referred to as “viewing angle”). Fig.11 This difference in the object audio data to be utilized due to the viewing angle when viewing is described. Note that although the horizontal viewing angle will be mainly described in the following description, the description for the vertical viewing angle will be similar.
[0178] Fig.11 This is an explanatory diagram for explaining the difference in target audio data to be used due to the difference in viewing angles during viewing. Fig.11 The display status V34 shown in Fig. 10B The display state V34 shown in FIG. Fig.11 The image displayed in the display state V34 is Fig. 10B The cut image D331 is shown, and the user takes a viewing angle θ a In this display state V34, as described above, the desired sounds come from two directions respectively, and the client 200 needs to select and obtain two pieces of object audio data.
[0179] In addition, Fig.11The image displayed in the display state V35 shown is the same as the image in the display state V34 and can be Fig. 10B The cropped image D331 is shown. Fig.11 In the display state V35 shown, the user is viewing the image with a viewing angle smaller than θ a The viewing angle θ c In the case of viewing the image at a small viewing angle as in the display state V35, from the user's point of view, the two singers H2 and H3 appear to be together, and it is considered that it is sufficient as long as the sound comes from one direction in the center. Therefore, in this case, the client 200 only needs to select and obtain one piece of object audio data.
[0180] In summary, the combination of object audio data to be utilized (acquired) by the client 200 may differ depending on the direction and the viewing angle of the cut image and the combination of the viewing angle when viewing. Note that even in the case where the decoded image is a so-called 360-degree image, in a manner similar to the above example, the combination of object audio data to be utilized may differ depending on the direction, the viewing angle, and the combination of the viewing angle of the image cut from the 360-degree image. In this example, by specifying the display state using these types of information, it is possible to specify the combination of object audio data to be utilized.
[0181] In this example, the object selection information for selecting the object audio data may include the following information.
[0182] - Orientation information about the direction in which the image should be cut
[0183] -Cut view angle information related to the view angle of the cut image
[0184] - Viewing angle information related to the viewing angle relative to the display image when viewed
[0185] - Utilization information related to the target audio data to be utilized
[0186] Note that there are cases where, among the above-mentioned information, the cut angle information and the viewing angle information will be collectively referred to simply as angle information.
[0187] The direction information may include, for example, information about the angle in the horizontal direction and information about the angle in the vertical direction. Alternatively, in addition to the information about the angle in the horizontal direction and information about the angle in the vertical direction, the direction information may also include information about the angle range relative to the shear direction vector specified according to the angle in the horizontal direction and the angle in the vertical direction. Alternatively, the direction information may include information about the angle range in the horizontal direction and information about the angle range in the vertical direction.
[0188] The cropping perspective information may include, for example, information on the horizontal perspective range and vertical perspective range of the cropped image. Alternatively, in the case where the horizontal perspective and vertical perspective of the cropped image in a display state that will serve as a reference are determined in advance as reference perspectives, the cropping perspective information may include information on the magnification range relative to the reference perspective.
[0189] For example, the viewing angle information may include information on the range of the horizontal viewing angle and the range of the vertical viewing angle relative to the display image. Alternatively, in the case where the horizontal viewing angle and the vertical viewing angle relative to the display image in the display state that becomes the reference are determined in advance as the reference viewing angle, the viewing angle information may include information on the range of the magnification relative to the reference viewing angle.
[0190] The utilization information may include information indicating all types of combinations of object audio data to be utilized. Alternatively, the utilization information may include information indicating differences in combinations of object audio data to be utilized relative to combinations of predetermined (default) object audio data determined in advance.
[0191] Note that the combination of the predetermined object audio data can be determined in advance regardless of whether the utilization information includes the combination of the predetermined object audio data. For example, in a display state where the corresponding object selection information is not prepared, by utilizing the predetermined object audio data, the time and work for preparing the object selection information so that all display states are represented are omitted.
[0192] Will refer to Fig.12 A method in which the processing unit 210 of the client 200 selects and acquires object audio data to be utilized based on the object selection information according to the above-described example is described. Fig.12 is a flowchart showing a method in which the client 200 selects and acquires object audio data to be utilized based on object selection information according to this example. Fig.12 In the illustrated example, it is assumed that a combination of predetermined object audio data is determined in advance, and a plurality of pieces of object selection information have been acquired by the client 200 and stored in the storage unit 240 , for example.
[0193] First, if Fig.12 As shown, the processing unit 210 acquires information of a shearing direction, a shearing angle of view, and a current viewing angle of view of a shearing image to be displayed by the client 200 (S102). Here, the information of the shearing direction and the shearing angle of view of the shearing image displayed by the client 200 may be determined by, for example, the control unit 220 that performs shearing processing, and may be provided from the control unit 220 to the processing unit 210. In addition, the information of the current viewing angle of view may be specified by the control unit 220 based on sensor data acquired by the sensor unit 250, and may be provided from the control unit 220 to the processing unit 210.
[0194] Subsequently, the processing unit 210 specifies object selection information corresponding to the current display state among the plurality of pieces of object selection information stored in the storage unit 240 based on the information acquired in step S102 .
[0195] For example, the processing unit 210 may specify a piece of object selection information that matches the information acquired in step S102 or includes the information acquired in step S102 in a range. Specifically, the direction information included in each piece of object selection information is compared with the cutting direction of the displayed cut image, and the object selection information having the direction information that matches the cutting direction of the displayed cut image or includes the cutting direction of the displayed cut image in a range is extracted. Then, the processing unit 210 compares the perspective information included in each piece of extracted object selection information with the information of the cutting perspective of the displayed cut image and the current viewing perspective. Then, the processing unit 210 may specify a piece of object selection information including the following perspective information, which includes the cutting perspective of the cut image and the current viewing perspective.
[0196] Note that if no object selection information matching or including the information acquired in step S102 is found in the object selection information stored in the storage unit 240, the processing unit 210 does not need to specify the object selection information in step S104.
[0197] In the case where the object selection information corresponding to the current display state is specified (S106: Yes), the processing unit 210 selects a combination of object audio data to be acquired based on the utilization information included in the specified object selection information (S108). For example, in the case where information indicating all types of combinations of object audio data to be utilized is included as utilization information in the object selection information, the processing unit 210 only needs to select the object audio data indicated by the utilization information as is. Alternatively, in the case where information indicating the difference of a combination of object audio data to be utilized relative to a predetermined combination of object audio data is included as utilization information in the object selection information, the processing unit 210 only needs to select the object audio data to be acquired based on a predetermined combination of object audio data.
[0198] Subsequently, the processing unit 210 acquires the object audio data selected in step S108 from the server 100. Note that a method for acquiring the selected object audio data will be described in detail later.
[0199] Meanwhile, in a case where object selection information corresponding to the current display state has not been specified ( S106 : No), the processing unit 210 acquires predetermined object audio data ( S112 ).
[0200] Note that in the above-described processing, the processing from steps S104 to S108 may be performed by, for example, the segment file selecting unit 213 , and the processing in steps S110 and S112 may be performed by, for example, the segment file acquiring unit 214 .
[0201] <3-2. Example of Object Selection Information Using Angles Between Objects (Example 1-2)>
[0202] An example of object selection information using a viewing angle has been described above as Example 1-1. Subsequently, an example of object selection information using an angle between objects will be described as Example 1-2. In Example 1-2 described below, the angle between two objects is calculated based on the positions of the two objects when viewed and the viewing position of the user, and object audio data to be utilized is selected based on the angle between the objects.
[0203] Fig.13 It is an illustration used to illustrate the angles between objects. Fig.13 As shown, in the display state V44, the angle between the object related to the target audio data AD2 (the object corresponding to the singer H2) and the object related to the target audio data AD3 (the object corresponding to the singer H3) is the angle θ. d At this angle θ d In the case where the difference between the positions of the two objects is small as in the display state V44, since the difference between the positions of the two objects will not be recognized by the user, it is sufficient that the sound comes from one direction of the center. Therefore, in the display state V44, the object audio data AD1 in which the object audio data AD2 and the object audio data AD3 are mixed is used.
[0204] On the other hand, Fig.13 As shown, in the display state V45, the angle between the object associated with the object audio data AD2 and the object associated with the object audio data AD3 is the angle θ e (θ e >θ d ). At this angle θ e In the case of being large as in the display state V45, since the user recognizes the positions of the two objects separately, it is expected that the sounds come from two directions, respectively. Therefore, in the display state V45, the object audio data AD2 and the object audio data AD3 are used.
[0205] As described above, the combination of object audio data to be utilized (acquired) by the client 200 may differ according to the size of the angle between two objects. In this example, the display state is specified according to the size of the angle between two objects, so that the combination of object audio data to be utilized can be specified.
[0206] In this example, the object selection information for selecting the object audio data may include the following information.
[0207] - The position information of the two objects between which the angle is to be calculated
[0208] - Angular information about the angle between two objects
[0209] - Utilization information related to the target audio data to be utilized
[0210] The position information of the two objects between which the angle is to be calculated may be similar information to the position information of the objects included in the above-mentioned object metadata. However, the two objects between which the angle is to be calculated are determined for each object selection information, and the position information of the two objects is included in the object selection information.
[0211] The angle information may include information such as an angle range. As will be described later, the angle information may be used as a threshold.
[0212] The utilization information according to this example may be similar to the utilization information in Example 1. That is, the utilization information may include information indicating all types of combinations of object audio data to be utilized. Alternatively, the utilization information may include information indicating the difference of the combination of object audio data to be utilized relative to the combination of predetermined object audio data determined in advance.
[0213] Will refer to Fig.14 A method in which the processing unit 210 of the client 200 selects and acquires object audio data to be utilized based on the object selection information according to the above-described example is described. Fig.14 is a flowchart showing a method in which the client 200 selects and acquires object audio data to be utilized based on object selection information according to this example. Fig.14 In the illustrated example, it is assumed that a combination of predetermined object audio data is determined in advance, and a plurality of pieces of object selection information have been acquired by the client 200 and stored in the storage unit 240 , for example.
[0214] First, if Fig.14 As shown, the processing unit 210 calculates the angle between the objects (S122). In step S122, the processing unit 210 may calculate the angle between the objects based on the position information of the two objects between which the angle is to be calculated, which is included in the object selection information stored in the storage unit 240, and the information of the viewing position of the user. In addition, the information of the viewing position of the user may be specified, for example, by the control unit 220 based on the sensor data acquired by the sensor unit 250, and may be provided from the control unit 220 to the processing unit 210.
[0215] Subsequently, the processing unit 210 specifies object selection information corresponding to the current display state among the plurality of pieces of object selection information stored in the storage unit 240 based on the angle between the objects calculated in step S122.
[0216] For example, the processing unit 210 extracts object selection information including angle information whose angle range includes the angle between the objects calculated in step S122. In addition, in the case where only one piece of object selection information is extracted, the processing unit 210 may specify the extracted piece of object selection information as the object selection information corresponding to the current display state. In the case where multiple pieces of object selection information are extracted, the processing unit 210 may specify the following object selection information among the multiple pieces of object selection information extracted, for which the positions of the two objects between which the angle is to be obtained are closest to the viewing direction of the user.
[0217] Note that, in the case where there is no object selection information including angle information whose angle range includes the angle between the objects calculated in step S122, the processing unit 210 does not have to specify the object selection information in step S124.
[0218] In the case where the object selection information corresponding to the current display state is specified (S126: Yes), the processing unit 210 selects a combination of object audio data to be acquired based on the utilization information included in the specified object selection information (S128). For example, in the case where information indicating all types of combinations of object audio data to be utilized is included as utilization information in the object selection information, the processing unit 210 only needs to select the object audio data indicated by the utilization information as is. Alternatively, in the case where information indicating the difference of a combination of object audio data to be utilized relative to a predetermined combination of object audio data is included as utilization information in the object selection information, the processing unit 210 only needs to select the object audio data to be acquired based on a predetermined combination of object audio data.
[0219] Subsequently, the processing unit 210 acquires the object audio data selected in step S128 from the server 100 .
[0220] If the object selection information corresponding to the current display state has not been specified ( S126 : No), the processing unit 210 acquires predetermined object audio data ( S120 ).
[0221] Note that in the above-described processing, the processing from steps S124 to S128 may be performed by, for example, the segment file selecting unit 213 , and the processing in steps S130 and S132 may be performed by, for example, the segment file acquiring unit 214 .
[0222] <<4. Example of considering documenting display status (Example 2)>>
[0223] Two examples (Example 1-1 and Example 1-2) have been described above as examples of object selection information and selection of object audio data using the object selection information. Subsequently, an example of filing object audio data in consideration of multiple display states will be described as Example 2. Note that Example 2 to be described below can be arbitrarily combined with the above examples, and can be combined with either Example 1-1 or Example 1-2.
[0224] In streaming using MPEG-DASH, in the case of acquiring object audio data, the object audio data is acquired for each audio file. By performing filing according to this example to be described below, it becomes possible to acquire the object audio data for each audio file, rather than acquiring the object audio data for each object. In addition, according to this example, the object audio data can be filed so that both the amount of data to be stored in the server 100 and the number of files to be acquired by the client 200 are suppressed.
[0225] In the following description, reference is made to Fig.15 After describing the documented process according to this example, we will refer to Figures 16 to 18 A specific example of filing is described. Note that filing of the target audio data described below is performed by the generation unit 110 of the server 100 , and may be performed by the segment file generation unit 113 of the generation unit 110 , for example.
[0226] Fig.15 is a flow chart showing the documentation process according to this embodiment. Fig.15 As shown, the segment file generation unit 113 groups the combination of the object audio data in a given display state (S202). In step S202, the combination of the object audio data in the display state may be specified based on, for example, the object selection information corresponding to the display state.
[0227] Subsequently, the segment file generating unit 113 groups the differences between the groups formed in step S202 and the combinations of the object audio data in other display states (display states other than the one display state used in step S202) (S204). In step S204, the segment file generating unit 113 groups the combinations of the object audio data to be newly used and the combinations of the object audio data not used when the display state is changed from the one display state used in step S202 to each display state, respectively and separately.
[0228] Subsequently, the segment file generation unit 113 selects a group including the least pieces of object audio data from among the groups formed in steps S202 and S204 (S206). Note that, in the case where there are a plurality of groups including the least pieces of object audio data, it is only necessary to select any one of the plurality of groups. In addition, as will be described later, the processing in step S206 may be repeatedly performed a plurality of times, and in the case where the processing in step S206 is performed a second time and a subsequent time, the segment file generation unit 113 selects a group including the least pieces of object audio data from among the unselected groups.
[0229] Subsequently, the segment file generating unit 113 removes the object audio data included in the group selected immediately before from the object audio data included in the unselected group ( S208 ).
[0230] In the case where there is an unselected group (S210: Yes), the process returns to step S206, and the process from step S206 to S210 is repeated until all groups are selected. If the process is repeated until there is no unselected group (S210: No), the segment file generation unit 113 files the selected groups per group (S212). In step S212, the segment file generation unit 113 generates an audio file by storing one or two or more pieces of object audio data included in each group in one audio file.
[0231] Note that the above-described processing from steps S202 to S212 may be performed by, for example, the segment file generating unit 113 .
[0232] The documentation process according to this example has been described above. Subsequently, a specific example of the above documentation will be described. Figures 16 to 18 It is an explanatory diagram for explaining a specific example of filing according to the present embodiment.
[0233] The following will describe the existence of Fig.16 The following are specific examples of files generated by the generation unit 110 in the three display states V51, V52 and V53 shown. Fig.16 As shown, in all display states V51, V52, and V53, images including four singers H11 to H14 are displayed. In addition, in the display state V51, three pieces of object audio data AD11, AD12, and AD15 are used. In addition, in the display state V52, four pieces of object audio data AD11, AD12, AD13, and AD14 are used. In addition, in the display state V53, one piece of object audio data AD16 is used.
[0234] Will refer to Fig.17 Description for Fig.16 The display states V51 to V53 shown perform Fig.15If the processing in step S202 is performed for display state V51, the operation can be performed as follows. Fig.17 Then, if the processing in step S204 is performed for the transition from the display state V51 to the display state V52 and the transition from the display state V51 to the display state V53, the group G11 can be obtained as shown in step S222 in FIG. Fig.17 Groups G12 to G15 are obtained as shown in step S224 in .
[0235] Will refer to Fig.18 Description for Fig.17 The five groups G11 to G15 shown perform Fig.15 First, before the first execution of the process in step S206, as shown in FIG. Fig.18 As shown in step S226 in FIG. 1 , the state is a state where all groups G11 to G15 are not selected. If the processing in step S206 is performed in the state of step S226, for example, group G12 is selected. In addition, if the processing in step S208 is performed, as shown in FIG. Fig.18 As shown in step S228 in , the target audio data AD15 included in the group G12 selected immediately before is removed from the groups G11 and G14. In the state in step S228, since there is an unselected group, it is determined as "yes" in step S210, and the processing in step S206 is performed again.
[0236] If the processing in step S206 is performed in the state of step S228, for example, group G15 is selected. Here, in the state of step S228, the object audio data AD16 included in group G15 is not included in other groups. Therefore, even if the processing in step S208 is performed, as shown in FIG. Fig.18 As shown in step S230 in FIG. 1 , the target audio data included in each group also does not change. In the state in step S230 , since there is an unselected group, “Yes” is determined in step S210 , and the process in step S206 is performed again.
[0237] If the processing in step S206 is performed in the state of step S230, for example, group G11 is selected. Here, in the state of step S230, the object audio data included in group G11 is the same as the object audio data included in group G14. Therefore, if the processing in step S208 is performed, as shown in FIG. Fig.18As shown in step S232 in , the object audio data AD11 and AD12 included in the group G11 selected immediately before are removed from the group G14, and the group G14 disappears. In the state of step S232, since there is an unselected group, it is determined as "yes" in step S210, and the processing in step S206 is performed again.
[0238] If the processing in step S206 is performed in the state of step S232, for example, group G13 is selected. Here, if group G13 is selected in the state of step S230, there is no unselected group. Therefore, even if the processing in step S208 is performed, Fig.18 As shown in step S234 in FIG. 1 , the target audio data included in each group also does not change. In the state of step S234 , since there is no unselected group, “No” is determined in step S210 , and the processing in step S212 is performed.
[0239] If the processing in step S212 is performed in the state of step S234, as shown in FIG. Fig.18 As shown in step S236 in FIG. 2 , audio files AF1 to AF4 are generated in which the object audio data respectively included in the groups G12 , G15 , G11 , and G13 are stored.
[0240] like Fig.18 As shown, since each of the object audio data AD11 to AD16 is stored in any one of the audio files AF1 to AF4, the server 100 does not need to store a plurality of identical object audio data. Therefore, the amount of object audio data to be stored in the server 100 can be minimized.
[0241] In addition, through Fig.18 By filing the target audio data as shown, it is possible to suppress Fig.16 The number of files to be acquired by the client 200 in each of the display states V51 to V53 shown. Fig.16 In the display state V51 shown, only two audio files, audio file AF1 and audio file AF3, need to be acquired. Fig.16 In the display state V52 shown, only two audio files, audio file AF3 and audio file AF4, need to be acquired. Fig.16 In the display state V53 shown, only one audio file, namely, the audio file AF2, needs to be acquired.
[0242] Among them, when a piece of object audio data is file-based so as to be stored in one file, three audio files need to be acquired in display state V51, four audio files need to be acquired in display state V52, and one audio file needs to be acquired in display state V51.
[0243] Therefore, by Fig.18 By filing the target audio data as shown, the number of audio files to be acquired in the display state V51 and the display state V52 is suppressed compared with the case where one piece of target audio data is filed to be stored in one file.
[0244] As described above, according to the present example, it is possible to generate an audio file so that both the amount of data to be stored in the server 100 and the number of files to be acquired by the client 200 are suppressed.
[0245] <<5. Example of Signaling Object Selection Information in a Case Where Combination of Object Audio Data Does Not Change Over Time (Example 3)>>
[0246] An example of filing object audio data in consideration of a plurality of display states has been described above as Example 2. Subsequently, an example of signaling object selection information in a case where a combination of object audio data to be utilized does not change over time will be described.
[0247] Note that the information to be transmitted with the signal may differ depending on whether the object selection information is information using the viewing angle described in Example 1-1 or information using the angle between objects described in Example 1-2. Therefore, in the following description, Example 3-1 corresponding to Example 1-1 and Example 3-2 corresponding to Example 1-2 will be described sequentially.
[0248] <3-1. Example of Signaling Object Selection Information Using Viewing Angle (Example 3-1)>
[0249] First, as Example 3-1, an example of signaling the object selection information using the perspective described in Example 1-1 will be described. In this example, when the utilization information described in Example 1-1 is signaled at the file level, and when streaming is implemented using MPEG-DASH, the utilization information only needs to be signaled at the AdaptationSet level.
[0250] Note that an example in which the object selection information includes the following information will be described below.
[0251] -Direction information, including horizontal angle information and vertical angle information
[0252] -Cut viewing angle information, including the horizontal viewing angle range information and the vertical viewing angle range information of the cut image
[0253] - Viewing angle information, including information about the horizontal viewing angle range and vertical viewing angle range relative to the displayed image when viewing
[0254] - Utilization information indicating all AdaptationSets including the object audio data to be utilized
[0255] In this example, information for display status is newly signaled by SupplementalProperty. schemeIdUri specifies "urn:mpeg:dash:objectAudio:objectSelection", and "direction_azimuth, direction_elevation, clipping_azimuth_range, clipping_elevation_range, viewing_azimuth_range, viewing_elevation_range, and AdaptationSet_list" are signaled by value in sequence separated by commas. Fig.19 is a table showing the values to be signaled in this example and a detailed description of each value.
[0256] In this example, the object selection information may be stored in the MPD file. Fig. 20 and Fig.21 An example of an MPD file to be generated by the MPD file generating unit 114 of the server 100 is described. Fig. 20 : is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 in the present embodiment. Fig.21 It is shown Fig. 20 The table of each piece of information in the MPD file shown. Fig.21 In the table, direction information, cutting angle information, viewing angle information, utilization information and Fig. 20 The corresponding SupplementalProperty in the MPD file shown.
[0257] exist Fig. 20 In the MPD file of , as shown in the third line, the combination of the predetermined (default) AdaptationSet in the preselectionComponents of Preselection is "o2 o3". Fig. 20In the MPD file of , four patterns of SupplementalProperty that are combinations different from the predetermined combination are shown from the fourth to seventh lines. In this way, by predetermining the combination of the predetermined AdaptationSet, it is only necessary to prepare SupplementalProperty only for the combination in which the object audio data to be utilized is different in each display state.
[0258] exist Fig. 20 In the example of the MPD file shown, reference is made to Fig. 22 A method in which the processing unit 210 of the client 200 selects an AdaptationSet to be acquired is described. Fig. 22 is a flowchart illustrating an example of a method for selecting an AdaptationSet acquired by the client 200 .
[0259] First, if Fig. 22 As shown, the processing unit 210 obtains information of a cutting direction, a cutting viewing angle, and a current viewing viewing angle of a cutting image to be displayed by the client 200 (S302).
[0260] Then, the processing unit 210 obtains all the SupplementalProperties of the Preselection whose schemeIdUri is “urn:mpeg:dash:objectAudio:objectSelection” ( S304 ).
[0261] Subsequently, the processing unit 210 specifies the SupplementalProperty corresponding to the current display state in the SupplementalProperty acquired in step S304 based on the information acquired in step S302 (S306). For example, the SupplementalProperty specified in step S306 may be a SupplementalProperty for which the values of direction_azimuth and direction_elevation are close to the clipping direction of the clipping image to be displayed by the client 200. In addition, the SupplementalProperty specified in step S306 may be a SupplementalProperty for which the clipping viewing angle of the clipping image to be displayed by the client 200 is included in clipping_azimuth_range and clipping_elevation_range. Furthermore, the SupplementalProperty specified in step S306 may be a SupplementalProperty for which the current viewing viewing angle is included in viewing_azimuth_range and viewing_elevation_range. Note that, in the case where there is no SupplementalProperty satisfying the above conditions, the processing unit 210 does not have to specify any SupplementalProperty in step S306.
[0262] In the case where the SupplementalProperty corresponding to the current display state is specified ( S308 : Yes), the processing unit 210 selects an AdaptationSet shown in the AdaptationSet_list of the specified SupplementalProperty ( S310 ).
[0263] If no SupplementalProperty corresponding to the current display state is specified ( S308 : No), the processing unit 210 selects a predetermined (default) AdaptationSet shown in Preselection@preselectionComponents ( S312 ).
[0264] Notice, Fig. 22 The illustrated processing from steps S302 to S312 may be performed by, for example, the segment file selection unit 213 .
[0265] An example of an MPD file according to this example has been described above. However, in this example, the MPD file to be generated is not limited to Fig. 20 Example shown.
[0266] For example, as a modification example of the MPD file in the case of using Preselection, the AdaptationSet_list of the value of SupplementalProperty may be signaled using preselectionComponents. Fig.23 is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 in the modified example. Fig.23 In the MPD file, as shown in the third, seventh and eighth lines, Preselection is separated, and the part that becomes AdaptationSet_list is shown as preselectionComponents. Fig.23 The rest of the MPD file is shown with Fig. 20 The MPD files shown are similar.
[0267] As another modification example, viewing_azimuth_range and viewing_elevation_range do not have to be included (set) in the MPD file. Fig. 22 In step S306, only direction_azimuth, direction_elevation, clipping_azimuth_range and clipping_elevation_range need to be used to select an AdaptationSet. However, in this case, since the viewing angle is not considered, the selected AdaptationSet may not be the most appropriate AdaptationSet.
[0268] As still another modification example, in addition to direction_azimuth and direction_elevation, direction_angle may be included as direction information in the MPD file. Here, direction_angle indicates an angle relative to the shear direction vector described in Example 1-1.
[0269] As still another modification example, instead of direction_azimuth and direction_elevation, direction_azimuth_range and direction_elevation_range may be included as direction information in the MPD file. Here, direction_azimuth_range indicates the angle range in the horizontal direction described in Example 1-1, and direction_elevation_range indicates the angle range in the vertical direction described in Example 1-1.
[0270] As a further modification example, instead of clipping_azimuth_range, clipping_elevation_range, viewing_azimuth_range, and viewing_elevation_range, clipping_magnification_range and viewing_magnification_range may be included as the clipping angle information and the viewing angle information. Here, clipping_magnification_range indicates the magnification range of the angle of view at the time of clipping described in Example 1-1, and viewing_magnification_range indicates the magnification range of the viewing angle of view described in Example 1-1.
[0271] As a further modification example, AdaptationSet_list may be replaced by a list of differences from preselectionComponents. For example, AdaptationSet_list may be replaced by AdaptationSet_del_list and AdaptationSet_add_list. AdaptationSet_del_list is a list of AdaptationSets to be deleted from preselectionComponents, and describes AdaptationSet@ids with AdaptationSet@ids separated by spaces. In addition, AdaptationSet_add_list is a list of AdaptationSets to be added to preselectionComponents, and describes AdaptationSet@ids with AdaptationSet@ids separated by spaces. Note that AdaptationSet_del_list and AdaptationSet_add_list correspond to the information indicating the difference between a combination of object audio data to be utilized and a combination of predetermined object audio data described in Example 1-1.
[0272] As a further modification example, SupplementalProperty can be transmitted in AdaptationSet by signaling. In the case where a combination of object audio data is stored in one file (in the case where the object audio data is not stored in multiple files), because Preselection is not used, by transmitting SupplementalProperty in AdaptationSet by signaling, the AdaptationSet to be used can be selected.
[0273] <5-2. Example of Signaling Object Selection Information Using Angles Between Objects (Example 3-2)>
[0274] An example of signaling object selection information using a viewing angle has been described above as Example 3-1. Next, as Example 3-2, an example of signaling object selection information using an angle between objects described in Example 1-2 will be described. In this example, in the case where the utilization information described in Example 1-2 is signaled at the file level and streaming is implemented using MPEG-DASH, the utilization information only needs to be signaled at the AdaptationSet level.
[0275] Note that an example in which the object selection information includes the following information will be described below.
[0276] - The position information of the two objects between which the angle is to be calculated
[0277] - Angle information including information on the angle range between objects, which becomes the threshold
[0278] - Utilization information indicating all AdaptationSets including the object audio data to be utilized
[0279] In this example, information for display status is newly signaled by SupplementalProperty. schemeIdUri specifies "urn:mpeg:dash:objectAudio:objectSelectionAngle", and "object1_azimuth, object1_elevation, object2_azimuth, object2_elevation, threshold_range, AdaptationSet_list" are sequentially signaled by value while being separated by commas. Fig.24 is a table showing the values to be transmitted by signal in this example and the detailed description of each value.
[0280] In this example, we will refer to Fig.25 and Fig.26 An example of an MPD file to be generated by the MPD file generating unit 114 of the server 100 is described. Fig.25 : is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 in the present embodiment. Fig.26 It is shown Fig.25 The table of each piece of information in the MPD file shown. Fig.26 The table shows the position information of the first object, the position information of the second object, the angle information, the utilization information and Fig.25 The corresponding SupplementalProperty in the MPD file shown.
[0281] exist Fig.25 In the MPD file, as shown in the third line, the combination of the predetermined (default) AdaptationSet in the preselectionComponents of Preselection is "o2 o3". In addition, Fig.25In the MPD file of , the fourth to fifth lines show two modes of SupplementalProperty as a combination different from the predetermined combination. In this way, by predetermining the combination of the predetermined AdaptationSet, it is only necessary to prepare SupplementalProperty only for the combination in which the object audio data to be used is different in each display state.
[0282] exist Fig.25 In the example of the MPD file shown, reference is made to Fig. 27 A method in which the processing unit 210 of the client 200 selects an AdaptationSet to be acquired is described. Fig. 27 is a flowchart illustrating an example of a method for selecting an AdaptationSet acquired by the client 200 .
[0283] First, if Fig. 27 As shown, the processing unit 210 obtains a list of AdaptationSets from Preselection@preselectionComponents (S322). Subsequently, the processing unit 210 obtains all of the SupplementalProperties of the Preselection whose schemeIdUri is "urn:mpeg:dash:objectAudio:objectSelectionAngle" (S324).
[0284] Then, the processing unit 210 calculates the angle between the objects when viewed for all SupplementalProperties acquired in step S324 (S326). In step S326, the processing unit 210 calculates the position of the object when viewed based on object1_azimuth, object1_elevation, object2_azimuth and object2_elevation in each SupplementalProperty, and calculates the angle between the objects when viewed.
[0285] Subsequently, the processing unit 210 specifies the SupplementalProperty corresponding to the current display state in the SupplementalProperty acquired in step S324 based on the angle between the objects calculated in step S326 (S328). For example, in step S328, the processing unit 210 may specify the SupplementalProperty for which the angle between the objects calculated in step S326 is included in the threshold_range in the SupplementalProperty acquired in step S324. Note that in the case where there is no SupplementalProperty that satisfies the above conditions, the processing unit 210 does not have to specify any SupplementalProperty in step S328.
[0286] In the case where the SupplementalProperty corresponding to the current display state is specified ( S330 : Yes), the processing unit 210 selects an AdaptationSet shown in the AdaptationSet_list of the specified SupplementalProperty ( S332 ).
[0287] If no SupplementalProperty corresponding to the current display state is specified ( S330 : No), the processing unit 210 selects a predetermined (default) AdaptationSet indicated in Preselection@preselectionComponents ( S334 ).
[0288] Notice, Fig. 27 The illustrated processing from steps S322 to S334 may be performed by, for example, the segment file selection unit 213 .
[0289] An example of an MPD file according to this example has been described above. However, in this example, the MPD file to be generated is not limited to Fig.25 Example shown.
[0290] For example, as a modification example, the location information of the object can be obtained from the metadata file. For example, in the case where there is a metadata file that stores only the object metadata including the location information of all objects, a link for accessing the metadata file (an example of metadata file access information) can be transmitted with a signal. Note that such a metadata file can be interpreted as a metadata file including (part of) object selection information. In addition, such a metadata file can be generated by, for example, the segment file generation unit 113 of the server 100, and stored in the storage unit 140 of the server 100.
[0291] Fig.28 is a view showing an example of an MPD file to be generated by the MPD file generating unit 114 in the modified example. Fig.28 In the MPD file, object1_azimuth, object1_elevation, object2_azimuth, and object2_elevation are not included. Fig.28 In the MPD file of , as shown in the fourth and fourteenth lines, a link to the metadata file (an example of metadata file access information) is stored in the SupplementalProperty of AdaptationSet. Fig.28 In the MPD file, as shown in the fifth and sixth lines, the two objects whose angles are to be calculated are indicated by the id (identifier) (object_metadata_AdaptationSet) of the metadata file and the two numbers (object1_num and object2_num) of the object metadata in the id.
[0292] The segment file acquisition unit 214 of the client 200 generates request information of the metadata file based on such a link to the metadata file to obtain the position information of the object, and acquires the metadata file from the server 100. Then, the segment file acquisition unit 214 acquires azimuth information and elevation information of the corresponding object as the position information from the metadata file.
[0293] As another modification example, instead of AdaptationSet_list, a list of AdaptationSets before replacement (before_AdaptationSet_list) and a list of AdaptationSets after replacement (after_AdaptationSet_list) may be included in the MPD file when the angle is included in the threshold_range. For example, this method may also be used when three or more pieces of object audio data are integrated (mixed) into one piece of object audio data.
[0294] In addition, although an example of calculating an angle based on the positions of two objects has been described above, angle information also including spread can be used by using spread information included in object metadata. In the case of performing signal transmission with the value of SupplementalProperty, spread information is added to each of object1 and object2. Specifically, spread_width, spread_height, and spread_radius can be included in the MPD file with spread_width, spread_height, and spread_radius separated by a space. If they are not separated by a space, only spread_width can be used.
[0295] In addition, as threshold_range, the MPD file may include a threshold in the horizontal direction (threshold_azimuth_range) and a threshold in the vertical direction (threshold_elevation_range), respectively. The processing unit 210 of the client 200 may use only the horizontal direction, only the vertical direction, or both the horizontal and vertical directions. In addition, information specifying whether to use only the horizontal direction, only the vertical direction, or both the horizontal and vertical directions may be included in the MPD file.
[0296] <<6. Example of Signaling Object Selection Information in a Case Where Combination of Object Audio Data Changes with Time (Example 4)>>
[0297] An example of signaling object selection information in the case where the combination of object audio data to be utilized does not change over time has been described above. Subsequently, an example of signaling object selection information in the case where the combination of object audio data to be utilized changes over time will be described. Note that, for example, whether the combination of object audio data to be utilized changes over time can be appropriately selected by the creator of the content.
[0298] In streaming using MPEG-DASH, a combination of object audio data to be used at each moment or a combination of required files can be stored in a content file and transmitted. Here, the content file can be, for example, an ISO Base Media File Format (ISOBMFF) file (an example of an MP4 file) whose standard is defined in ISO / IEC 14496-12.
[0299] In the case where the combination of object audio data to be utilized changes over time, the client 200 only needs to pre-acquire the ISOBMFF file and determine the file to be acquired according to time. In the following description, seven examples (Example 4-1 to Example 4-7) of signal transmission of object selection information in the case where the combination of object audio data changes over time will be described. In Example 4-1 to Example 4-7 described below, an example of storing object selection information in a metadata file included in an ISOBMFF file will be described.
[0300] <6-1. Example of extended metadata file (Example 4-1)>
[0301] First, as Example 4-1, an example in which the segment file generating unit 113 generates an extended metadata file will be described.
[0302] First, refer to Fig.29 and Fig.30 An example of the structure of a metadata file before being expanded is described. Fig.29 and 30 This is a diagram for explaining the structure of a metadata file. Fig.30 As shown, Fig.29 The illustrated audio file 1 (file1) includes object audio data 1 (obj1) and object audio data 2 (obj2) In the audio file 1, streamID as a file identifier is stored in MHAMultiStreamBox ('maeM'), and streamID indicates "1". Fig.29 The object audio file 2 (file2) shown in FIG. 1 includes Fig.30 The object audio data 3 (obj3) is shown, and the streamID indicates "2".
[0303] Fig.29 The metadata file shown in FIG. 1 includes the object metadata included in the audio file 1 and the audio file 2. Fig.30As shown, reference_streamID of sampleEntry('a3am') of the metadata file (metadata file) indicates which file includes object metadata. In the metadata file, sampleEntry(sample entry) indicates object metadata (num_reference_streamID=2) including two files, the first object metadata included in sample(sample) is the object metadata of the file whose streamID=1(reference_streamID=1), and the second object metadata is the object metadata of the file whose streamID=2(reference_streamID=2). Fig.31 It is shown Fig.29 and Fig.30 A view of the syntax of the metadata file is shown.
[0304] Note that MHAMultiStreamBox has Fig.61 As shown in the structure, 3da_meta_data() has Fig.62 The structure shown in the figure, and DSE has Fig.63 The structure shown in . Note that Fig.63 The 3da_ancillary_data stored in the data_stream_byte in the DSE shown in FIG. Fig.64 However, in the case where the size of 3da_meta_data() is larger than the maximum size of data_stream_byte of DSE, 3da_meta_data() is divided and stored in a plurality of DSEs.
[0305] In this example, the extended reference Figure 29 to Figure 31 In this example, the segment file generation unit 113 generates an extended metadata file in which object selection information for selecting object audio data is stored. Fig.32 and Fig.33 Describes an example of the extended metadata file to be generated in this example.
[0306] Fig.32 is a view showing an example of an extended metadata file according to this example. Fig.33 is a view showing the syntax of the extended metadata file according to the present example.
[0307] like Fig.32 and Fig.33As shown in FIG. 1 , in the extended metadata file, objectSelectionMetadataConfigurationBox() is additionally stored in SampleEntry('a3am') of the moov box, and objectSelectionMetadata() is additionally stored in sample of mdat. Object selection information may be stored in such objectSelectionMetadataConfigurationBox() and objectSelectionMetadata(). In addition, as Fig.33 As shown in FIG. 1 , in the extended metadata file, the object selection metadata () shows the combination of the object audio data or files to be used at each time. Fig.33 As shown, in the extended metadata file, objectSelectionMetadataConfigurationBox() stores information indicating in which file the object audio data is stored, which is indicated by objectSelectionMetadata(). Note that the details of objectSelectionMetadataConfigurationBox() and objectSelectionMetadata() will be described later.
[0308] Note that for MPEG-H 3D Audio metadata files, a similar reference can be performed. Fig.32 and Fig.33 Description metadata file extension.
[0309] <6-2. Example of object selection metadata file (Example 4-2)>
[0310] The extended metadata file has been described above as Example 4-1. Fig.34 and Fig.35 An example of generating a metadata file dedicated to object selection (hereinafter, referred to as "object selection metadata file") is described. In this example, the object selection metadata file may be generated by the segment file generating unit 113, and may include only object selection information.
[0311] Fig.34 is a view showing an example of selecting a metadata file according to the object of this example. Fig.35 is a view showing the syntax of the object selection metadata file according to this example.
[0312] like Fig.34 and Fig.35As shown, in the object selection metadata file, objectSelectionMetadataConfigurationBox() is stored in the SampleEntry of the moov box in a similar manner to the above-mentioned extended metadata file. Fig.34 and Fig.35 As shown, in the object selection metadata file, objectSelectionMetadata() is stored in sample of mdat in a similar manner to the above-mentioned extended metadata file. Note that the details of objectSelectionMetadataConfigurationBox() and objectSelectionMetadata() will be described later.
[0313] Note that references can be generated similarly in MPEG-H 3D Audio. Fig.34 and Fig.35 A metadata file describing the object selection.
[0314] <6-3. Storage Example of Object Selection Information Using Viewpoint (Example 4-3)>
[0315] Subsequently, as Example 4-3, an example in which the object selection information using the perspective described in Example 1-1 is stored in the above metadata file will be described. This example can be combined with any of the above Examples 4-1 and 4-2. In the following description, objectSelectionMetadataConfigurationBox and objectSelectionMetadata according to this example will be described.
[0316] Fig.36 is a view showing an example of objectSelectionMetadataConfigurationBox according to this example. In addition, Fig.37 It is shown Fig.36 The table shows the value of objectSelectionMetadataConfigurationBox and the detailed description of each value. Fig.37 As shown, in Fig.36In the objectSelectionMetadataConfigurationBox shown, stream_id is used to indicate the audio file including the object_id shown in the objectSelectionMetadata, and object_num_in_stream is used to indicate the object audio data of which number the object audio data is. In this example, streamID (stream_id) is information for accessing the audio file in which the object audio data is stored (hereinafter, also referred to as "audio file access information"). In addition, default_object_id is used to indicate the predetermined (default) object audio data to be used.
[0317] Fig.38 is a view showing an example of objectSelectionMetadata according to this example. In addition, Fig.39 It is shown Fig.38 The table shows the value of objectSelectionMetadata and the detailed description of each value. Fig.38 In the objectSelectionMetadata shown, the number of SupplementalProperties in Example 3-1 is indicated by mixed_object_list_num. In addition, regarding other parameters, the part of the AdaptationSet list shown in Example 3-1 is indicated by a list of ids (object_id) of object audio data (hereinafter, also referred to as "object list"). The combination of object audio data to be acquired can be indicated by combining this example with the above-mentioned Example 4-1 or Example 4-2.
[0318] Note that objectSelectionMetadata according to this example is not limited to Fig.38 and Fig.39 For example, as a modified example, instead of Fig.38 The fourteenth to seventeenth lines of the objectSelectionMetadata shown may show the difference from the default object (defaultobject) indicated by the objectSelectionMetadataConfigurationBox. Fig.40 is a view showing a modified example of such objectSelectionMetadata. Note that Fig.40 Only the available Fig.38 The fourteenth to seventeenth lines of the objectSelectionMetadata shown are replaced. In addition, Fig.41 It is shown Fig.40 A table with detailed descriptions of each value shown.
[0319] As another modification example, in addition to direction_azimuth and direction_elevation, direction_angle may be included in objectSelectionMetadata as direction information. Here, direction_angle indicates an angle relative to the shear direction vector described in Example 1-1.
[0320] As a further modification example, as direction information, min_direction_azimuth_range, max_direction_azimuth_range, min_direction_elevation_range, and max_direction_elevation_range may be included in objectSelectionMetadata instead of direction_azimuth and direction_elevation. Here, min_direction_azimuth_range and max_direction_azimuth_range indicate the angle range in the horizontal direction described in Example 1-1, and min_direction_elevation_range and max_direction_elevation_range indicate the angle range in the vertical direction described in Example 1-1.
[0321] As another modification example, as the clipping angle information and the viewing angle information, min_clipping_magnification_range, max_clipping_magnification_range, min_viewing_magnification_range, and max_viewing_magnification_range may be included in objectSelectionMetadata instead of min_clipping_azimuth_range, max_clipping_azimuth_range, min_clipping_elevation_range, and max_clipping_elevation_range. Here, min_clipping_magnification_range and max_clipping_magnification_range indicate the magnification range of the angle of view at the time of clipping described in Example 1-1, and min_viewing_magnification_range and max_viewing_magnification_range indicate the magnification range of the viewing angle described in Example 1-1.
[0322] As a further modification example, in objectSelectionMetadataConfigurationBox and objectSelectionMetadata, a combination of files may be shown instead of a combination of object audio data. Figure 42 to Figure 45 Such a modification example is described.
[0323] Fig.42 is a view showing an example of objectSelectionMetadataConfigurationBox according to this example. In addition, Fig.43 It is shown Fig.42 The table shows the value of objectSelectionMetadataConfigurationBox and the detailed description of each value. Fig.42The objectSelectionMetadataConfigurationBox shown does not include information indicating in which file the object audio data is included and information of the predetermined object audio data to be used. Fig.42 In the objectSelectionMetadataConfigurationBox shown, stream_id is used as default_stream_id to indicate a combination of audio files to be used in a predetermined (default) state. Fig.44 is a view showing an example of objectSelectionMetadata according to the present modification example. Fig.45 It is shown Fig.44 The table shows the value of objectSelectionMetadata and the detailed description of each value. Fig.44 In the objectSelectionMetadata shown, stream_id is used to indicate a combination of audio files, not a combination of object audio data. Also in this example, stream_id is audio file access information for accessing an audio file in which object audio data is stored.
[0324] According to reference Figure 42 to Figure 45 The present example described provides the following advantage: the client 200 does not need to determine the file including the target audio data to be acquired.
[0325] <6-4. Storage Example of Object Selection Information Using Angles Between Objects (Example 4-4)>
[0326] An example of storing object selection information using a viewing angle has been described above as Example 4-3. Subsequently, as Example 4-4, an example in which the object selection information using an angle between objects described in Example 1-2 is stored in the above metadata file will be described. This example can be combined with any of the above Examples 4-1 and 4-2.
[0327] In this example, objectSelectionMetadataConfigurationBox may be similar to the objectSelectionMetadataConfigurationBox in the above-mentioned Example 4-3. In the following description, objectSelectionMetadata according to this example will be described.
[0328] Fig.46 is a view showing an example of objectSelectionMetadata according to this example. In addition, Fig.47 It is shown Fig.46 The table shows the value of objectSelectionMetadata and the detailed description of each value. Fig.46 In the objectSelectionMetadata shown, mixed_object_list_num is used to indicate the number of SupplementalProperties in Example 3-2. In addition, regarding other parameters, a list of ids (object_id) of object audio data is used to indicate the part of the AdaptationSet list shown in Example 3-2. The combination of object audio data to be obtained can be indicated by combining this example with the above-mentioned Example 4-1 or Example 4-2.
[0329] Note that objectSelectionMetadata according to this example is not limited to Fig.46 and Fig.47 For example, as a modified example, instead of Fig.46 The tenth to thirteenth lines of the objectSelectionMetadata shown may include a list of objects before replacing a default object shown in the objectSelectionMetadataConfigurationBox and a list of objects after replacing the default object. Fig.48 is a view showing a modified example of such objectSelectionMetadata. Note that Fig.48 Only the available Fig.46 The 10th to 13th lines of the objectSelectionMetadata shown are replaced. In addition, Fig.49 It is shown Fig.48 A table with detailed descriptions of each value shown.
[0330] As another modification example, the location information of the object can be obtained from the object metadata. For example, instead of object1_azimuth, object1_elevation, object2_azimuth, and object2_elevation, the objectSelectionMetadata according to this modification example may include object1_id and object2_id. In addition, the processing unit 210 of the client 200 can obtain azimuth (azimuth) and elevation (elevation) from the object metadata. In the case where objectSelectionMetadata is not included in the object metadata, it is only necessary to extend the track reference (track reference), set reference_type'obmt', specify the track of the object metadata, and perform signal transmission so that the object metadata on the track can be utilized.
[0331] <6-5. Example of Signal Transmission Enabling Selection of Object Selection Information (Example 4-5)>
[0332] Subsequently, as Example 4-5, an example of implementing various structures by including additional information in the objectSelectionMetadataConfigurationBox described in Example 4-3 and Example 4-4 is described. For example, in this example, a signal indicating that the object audio data is used as reference Fig.36 Describes the list of audio files in the object to indicate or use as a reference Fig.42 Describes the list of files (streams) to indicate the information.
[0333] Fig.50 is a view showing an example of objectSelectionMetadataConfigurationBox according to this example. In addition, Fig.51 It is shown Fig.50 A table showing the value of objectSelectionMetadataConfigurationBox and a detailed description of each value.
[0334] Fig.50 The objectSelectionMetadataConfigurationBox shown includes list_is_object and sample_type, where list_is_object indicates that the Fig.36 and Fig.42The flag of which type of objectSelectionMetadataConfigurationBox is set, and sample_type indicates the structure of objectSelectionMetadata. According to such configuration, various structures can be implemented.
[0335] <6-6. Example of signal transmission through MPD file (Example 4-6)>
[0336] Subsequently, an example of signal transmission of an ISOBMFF file including object selection information through an MPD file will be described as Examples 4-6. The MPD file described in this example can be generated by the MPD file generation unit 114 of the server 100. In addition, in the MPD file generated in this example, metadata file access information for accessing a metadata file including object selection information is stored.
[0337] First, refer to Fig.52 An example of an MPD file generated by the MPD file generating unit 114 of the server 100 in a case where object selection information is stored in the extended metadata file described above as Example 4-1 is described. Fig.52 is a view showing an example of an MPD file to be generated according to the present example.
[0338] exist Fig.52 In the MPD file, as shown in the third line, schemeIdUri specifies urn:mpeg:dash:objectAudio:objectMetadataFile in the SupplementalProperty of Preselection, and uses value to specify the AdaptationSet@id of the metadata file. In addition, Fig.52 As shown in the twelfth line of the MPD file in , in the AdaptationSet including the metadata file, schemeIdUri specifies urn:mpeg:dash:objectAudio:objectMetadataStreamID in SupplementalProperty, so that the relationship between streamID and AdaptationSet can be indicated.
[0339] Next, we will refer to Fig.53 An example of an MPD file generated by the MPD file generating unit 114 of the server 100 in a case where object selection information is stored in the object selection metadata file described above as Example 4-2 is described. Fig.53 is a view showing an example of an MPD file to be generated according to the present example.
[0340] exist Fig.53 In the MPD file, as shown in the fourth line, schemeIdUri specifies urn:mpeg:dash:objectAudio:objectSelectionMetadataFile in the SupplementalProperty of Preselection, and uses value to specify the AdaptationSet@id of the object selection metadata file. Fig.53 In the MPD file, AdaptationSet@id="m2" is shown. Fig.53 In the MPD file of , as shown in the nineteenth line, in the AdaptationSet (AdaptationSet@id="m2") of the object selection metadata file, schemeIdUri is signaled in SupplementalProperty as urn:mpeg:dash:objectAudio:objectMetadataFile. In the case of "storing object selection information of the angle between two objects when using display" in objectSelectionMetadata, this is required in the case where it is necessary to refer to the metadata file to obtain the angle between the two objects. The signaling of urn:mpeg:dash:objectAudio:objectMetadataStreamID by schemeIdUri in SupplementalProperty is used to indicate which AdaptationSet file is indicated by the stream_id signaled in the object selection metadata file.
[0341] Note that the signaling of the urn:mpeg:dash:objectAudio:objectSelectionMetadataFile through the schemeIdUri in the SupplementalProperty of the Preselection can also be applied to the object selection information stored in the reference Fig.52 Describes the situation in the extended metadata file.
[0342] In addition, in the AdaptationSet (AdaptationSet@id="m2") of the object selection metadata file, the part where the schemeIdUri is signaled in the SupplementalProperty urn:mpeg:dash:objectAudio:objectMetadataFile can be indicated by the associationId of Representation. Fig.54 Such an example is described. Fig.54 is a view showing another example of an MPD file to be generated according to the present example.
[0343] exist Fig.54 In the MPD file shown, instead of Fig.53 In the twenty-first line of the MPD file shown<Representation id=“op2”> , showing<Representation id="op2"associationId=“op1”> .exist Fig.54 In the MPD file shown, schemeIdUri is not signaled using urn:mpeg:dash:objectAudio:objectMetadataFile in the SupplementalProperty (AdaptationSet@id="m2") of the AdaptationSet of the object selection metadata file.
[0344] <6-7. Example of selecting a file to be obtained by the client (Example 4-7)>
[0345] Subsequently, a method for selecting a file to be acquired by the processing unit 210 of the client 200 will be described as Examples 4-7. Fig.55 is a flowchart showing an example of a method for selecting a file to be acquired by a client. Fig.55 As shown in reference Figure 36 to Figure 39 Store object selection information in the reference Figure 32 to Figure 33 Described in the extended metadata file and using the reference Fig.52 An example of a selection method in the case where an MPD file is described to perform signal transmission.
[0346] First, if Fig.55As shown, the processing unit 210 specifies the AdaptationSet of the metadata file by using schemeIdUri from the value of "urn:mpeg:dash:objectAudio:objectMetadataFile" in the SupplementalProperty of the Preselection of the MPD file to obtain the metadata file from the server 100 (S402). Subsequently, the processing unit 210 obtains the information of the cutting direction, cutting angle and current viewing angle of the cut image displayed by the client 200 (S404). Subsequently, the processing unit 210 obtains the current objectSelectionMetadata from the metadata file obtained in step S402 (S406).
[0347] In addition, the processing unit 210 specifies an object list corresponding to the current display state based on the objectSelectionMetadata obtained in step S406 (S408). For example, the object list specified in step S408 may be an object list in which the clipping direction of the clipping image displayed by the client 200 is close to the values of direction_azimuth and direction_elevation. In addition, the object list specified in step S408 may be an object list in which the clipping viewing angle of the clipping image displayed by the client 200 is included in the viewing angle range indicated by min_clipping_azimuth_range, max_clipping_azimuth_range, min_clipping_elevation_range, and max_clipping_elevation_range. Further, the object list specified in step S408 may be an object list in which the current viewing angle is included in the viewing angle range indicated by min_viewing_azimuth_range, max_viewing_azimuth_range, min_viewing_elevation_range, and max_viewing_elevation_range. Note that, in the case where there is no object list satisfying the above conditions, the processing unit 210 does not have to specify any object list in step S408.
[0348] If the object list corresponding to the current display state is specified (S410: Yes), the processing unit 210 obtains the object_id of the specified object list (S412). If the object list corresponding to the current display state is not specified (S410: No), the processing unit 210 obtains the default_object_id of sampleEntry (S414).
[0349] Then, the processing unit 210 acquires the stream_id of the stream to which each object_id belongs from the objectSelectionMetadataConfigurationBox of sampleEntry (S416). In addition, the processing unit 210 selects the AdaptationSet (file) to be acquired based on the stream_id of urn:mpeg:dash:objectAudio:objectMetadataStreamID (S418).
[0350] Notice, Fig.55 The illustrated processing from steps S402 to S418 may be performed by, for example, the segment file selection unit 213 .
[0351] <<7. Application Examples>>
[0352] The embodiments of the present disclosure have been described above. The above-mentioned technology in the present disclosure (the present technology) can be applied to various types of content, and for example, can also be applied to multi-view (Multi View) content that can be viewed at multiple viewing positions. In the following description, an application example in which the present technology is applied to multi-view content will be described.
[0353] Figure 56 to Figure 58 is an explanatory diagram for explaining an application example. In the following description, the following will be described. Fig.56 As shown, there are three pieces of object audio data AD21 to AD23 and there are two viewing positions VP1 and VP2.
[0354] like Fig.57 As shown, in the case where the image is viewed at the viewing position VP1, the object audio data AD21 and the object audio data AD22 appear to overlap each other. Therefore, the object audio data AD24 in which the object audio data AD21 and the object audio data AD22 are integrated is prepared. Then, depending on the display state, there may be a case where three pieces of object audio data AD21 to AD23 are used, and a case where two pieces of object audio data AD23 and AD24 are used.
[0355] On the other hand, Fig.58As shown, in the case where the image is viewed at the viewing position VP2, the object audio data AD22 and the object audio data AD23 appear to overlap each other. Therefore, the object audio data AD25 in which the object audio data AD22 and the object audio data AD23 are integrated is prepared. Then, depending on the display state, there may be a case where three pieces of object audio data AD21 to AD23 are used, and a case where two pieces of object audio data AD22 and AD25 are used.
[0356] As reference Figure 56 to Figure 58 As described above, there may be a case where the object audio data to be utilized is different depending on the viewing position. In this case as well, in a manner similar to the above-described embodiment, it is only necessary to specify the object audio data to be utilized for each viewing position. Fig.59 An example of an MPD file to be generated in this case is described. Fig.59 is a view showing an example of an MPD file to be generated in a case where the target audio data to be utilized differs depending on the viewing position. Fig.59 An example of an MPD file to be generated when target audio data to be utilized does not change over time is described in .
[0357] exist Fig.59 In the example shown in , a viewing position is identified using Viewpoint, an existing element utilized in multi-view content. Each Viewpoint indicates a combination of AdaptationSets for each display state. view1 corresponding to the viewing position VP1 is indicated by Preselection@id="1", and the combination of predetermined object audio data is o1, o2, and o3, and o3 and o4 are shown as other types of combinations in SupplementalProperty. view2 corresponding to the viewing position VP2 is indicated by Preselection@id="2", and the combination of predetermined object audio data is o1, o2, and o3, and o2 and o5 are shown as other types of combinations in SupplementalProperty.
[0358] In addition, the above method can also be applied to free viewpoint content. Free viewpoint content is content that can be viewed from a free position within a given space. First, multi-view content is prepared, and a combination of object audio data to be utilized is determined at multiple viewing positions. In this case, each viewing position is indicated by using the position coordinates transmitted by the signal using the value of the Viewpoint. When reproducing, it is only necessary to select the Preselection of the Viewpoint that is closest to the desired viewpoint position among the viewpoint positions included in the Viewpoint, and determine the object to be utilized based on the meta-information in the Preselection.
[0359] <<8. Hardware Configuration Example>>
[0360] The embodiments of the present disclosure have been described above. Fig.60 The hardware configuration of the information processing apparatus according to the present embodiment is described. Fig.60 is a block diagram showing an example of the hardware configuration of the information processing device according to the present embodiment. Note that Fig.60 The information processing device 900 shown can implement, for example, Figure 8 and Fig. 9 The server 100 and the client 200 shown in FIG. Information processing performed by the server 100 and the client 200 according to the present embodiment is realized by cooperation of software and hardware described below.
[0361] like Fig.60 As shown, the information processing device 900 includes: a central processing unit (CPU) 901, a read-only memory (ROM) 902, a random access memory (RAM) 903, and a host bus 904a. In addition, the information processing device 900 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. Instead of the CPU 901 or in addition to the CPU 901, the information processing device 900 may include a processing circuit such as a DSP or an ASIC.
[0362] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the information processing apparatus 900 according to various programs. In addition, the CPU 901 may be a microprocessor. The ROM 902 stores programs, arithmetic parameters, etc. used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately in the execution, etc. The CPU 901 may form, for example, the generation unit 110, the control unit 120, the processing unit 210, and the control unit 220.
[0363] The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a host bus 904a including a CPU bus, etc. The host bus 904a is connected to an external bus 904b such as a peripheral component interconnect / interface (PCI) bus via a bridge 904. The host bus 904a, the bridge 904, and the external bus 904b are not necessarily configured separately, and these functions may be implemented on one bus.
[0364] For example, the input device 906 is implemented by a device to which a user inputs information, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a joystick. In addition, the input device 906 may be, for example, a remote control device using infrared or other radio waves, or may be an external connection device such as a mobile phone or a PDA corresponding to the operation of the information processing device 900. In addition, the input device 906 may include, for example, an input control circuit that generates an input signal based on information input by the user using the input means as described above and outputs the generated input signal to the CPU 901. The user of the information processing device 900 can input various data to the information processing device 900 and give instructions on processing operations by operating the input device 906.
[0365] The output device 907 is formed by a device capable of visually or auditorily notifying the user of the acquired information. As such a device, there are display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices or lamps, sound output devices such as speakers and headphones, printer devices, etc. The output device 907 outputs, for example, the results obtained by various processes performed by the information processing device 900. Specifically, the display device visually displays the results obtained by various processes performed by the information processing device 900 in various formats such as text, images, tables, and graphs. The sound output device converts an audio signal composed of reproduced audio data, acoustic data, etc. into an analog signal and outputs it auditorily. The output device 907 can be formed, for example, by the display unit 260 and the speaker 270.
[0366] The storage device 908 is a device for data storage, which is formed as an example of a storage unit of the information processing device 900. The storage device 908 is implemented by, for example, a magnetic storage unit device such as an HDD, a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage device 908 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, etc. The storage device 908 stores programs and various types of data executed by the CPU 901, various types of data acquired from the outside, etc. The storage device 908 may form, for example, the storage unit 140 and the storage unit 240.
[0367] The drive 909 is a reader / writer for a storage medium, and is built in the information processing apparatus 900 or externally attached to the information processing apparatus 900. The drive 909 reads information recorded in a removable storage medium such as a mounted magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 903. In addition, the drive 909 can write information in a removable storage medium.
[0368] The connection port 911 is an interface connected to an external device, and serves as a connection port to an external device capable of transmitting data through a universal serial bus (USB), for example.
[0369] The communication device 913 is a communication interface formed by, for example, a communication device for connecting to the network 920, etc. For example, the communication device 913 is a communication card for a wired or wireless local area network (LAN), long term evolution (LTE) LTE, Bluetooth (registered trademark) or wireless USB (WUSB), etc. In addition, the communication device 913 may be a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), various communication modems, etc. The communication device 913 may send and receive signals, etc., for example, on the Internet or with other communication devices according to a predetermined protocol such as TCP / IP. The communication device 913 may form, for example, the communication unit 130 and the communication unit 230.
[0370] For example, the sensor 915 corresponds to various types of sensors, such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, a light sensor, a sound sensor, a distance measurement sensor, and a force sensor. The sensor 915 acquires information related to the state of the information processing device 900, such as the posture of the housing of the information processing device 900, and information related to the environment around the information processing device 900, such as the luminosity and noise around the information processing device 900. In addition, the sensor 915 may include a GPS sensor that receives a GPS signal and measures the latitude, longitude, and altitude of the device. The sensor 915 may form, for example, the sensor unit 250.
[0371] The network 920 is a wired or wireless transmission path of information sent from a device connected to the network 920. For example, the network 920 may include: a public circuit network such as the Internet, a telephone circuit network, or a satellite communication network; various local area networks (LANs) including Ethernet (registered trademark); a wide area network (WAN), etc. The network 920 may also include a dedicated line network such as an Internet Protocol-Virtual Private Network (IP-VPN).
[0372] As described above, an example of a hardware configuration capable of realizing the functions of the information processing device 900 according to the present embodiment of the present disclosure has been described. The above components may be realized using general-purpose components, or may be realized by hardware dedicated to the functions of each component. Therefore, the hardware configuration to be used may be appropriately changed according to the technical level when executing the present embodiment of the present disclosure.
[0373] Note that a computer program for realizing each function of the information processing device 900 according to the present embodiment described above in accordance with the present disclosure may be prepared and implemented on a PC or the like. In addition, a computer-readable recording medium having such a computer program stored thereon may also be provided. For example, the recording medium is a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. In addition, the above-mentioned computer program may be distributed via, for example, a network without using any recording medium.
[0374] <<9. Conclusion>>
[0375] As described above, according to an embodiment of the present disclosure, object audio data can be selected according to the display state. For example, when the object audio data is transmitted using MPEG-DASH streaming, the sound can be heard from a more appropriate position suitable for the viewing environment. In addition, in the case where the display state is changed by zooming in or out, the user can have an experience in which the sound comes from an appropriate position according to the image at that time. Furthermore, the object audio data can be filed so that both the amount of data to be stored in the server and the number of files to be obtained by the client are suppressed. Therefore, the transmission amount is reduced, making it possible to increase the bit rate of other data to be obtained, for example.
[0376] Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the technical scope of the present invention is not limited to such examples. It will be obvious to those skilled in the art of the present invention that various modifications and variations can be conceived within the scope of the technical concept described in the claims, and such modifications and variations naturally fall within the technical scope of the present invention.
[0377] In addition, the effects described in this specification are merely illustrative or exemplary effects, and are not restrictive. That is, in addition to or in place of the above effects, the technology according to the present disclosure may also exhibit other effects that are obvious to those skilled in the art based on the description of this specification.
[0378] Note that the following configurations are also within the technical scope of the present disclosure. (1)
[0380] An information processing device, comprising:
[0381] a generating unit configured to generate a management file storing information related to object selection information for selecting object audio data according to a display state of an image; and
[0382] A sending unit is configured to send the management file. (2)
[0384] The information processing device according to (1), wherein the information related to the object selection information includes the object selection information. (3)
[0386] The information processing device according to (1) or (2), wherein the generation unit generates a metadata file including the object selection information, and
[0387] The information related to the object selection information includes metadata file access information for accessing the generated metadata file. (4)
[0389] The information processing device according to any one of (1) to (3), wherein the management file is a media presentation description (MPD) file. (5)
[0391] The information processing device according to (3), wherein the management file is a Media Presentation Description (MPD) file, and
[0392] The metadata file access information is stored in the AdaptationSet of the MPD file. (6)
[0394] The information processing device according to (5), wherein the metadata file access information is stored in a SupplementalProperty of the AdaptationSet. (7)
[0396] The information processing device according to any one of (1) to (6), wherein the object selection information includes direction information regarding a direction in which the image is cut out. (8)
[0398] The information processing device according to (7), wherein the direction information includes information on an angle in a horizontal direction and information on an angle in a vertical direction. (9)
[0400] The information processing device according to (8), wherein the direction information further includes information on an angle range relative to a shear direction vector. (10)
[0402] The information processing device according to (7), wherein the direction information includes information on an angle range in a horizontal direction and information on an angle range in a vertical direction. (11)
[0404] The information processing device according to any one of (1) to (10), wherein the object selection information includes viewing angle information related to a viewing angle of a cutout image or a viewing angle when viewed relative to a displayed image. (12)
[0406] The information processing device according to (11), wherein the viewing angle information includes information on a horizontal viewing angle range and information on a vertical viewing angle range. (13)
[0408] The information processing device according to (11), wherein the angle of view information includes information on a magnification range relative to a reference angle of view. (14)
[0410] The information processing device according to any one of (1) to (6), wherein the object selection information includes position information of two objects between which an angle is to be calculated. (15)
[0412] The information processing device according to (14), wherein the object selection information further includes angle information regarding an angle between the two objects. (16)
[0414] The information processing device according to any one of (1) to (15), wherein the object selection information includes information related to the object audio data to be utilized. (17)
[0416] The information processing device according to any one of (1) to (16), wherein the generation unit generates the audio file based on the combination of the object audio data according to the display state so that the same object audio data is not included in a plurality of audio files. (18)
[0418] An information processing method to be performed by an information processing device, the information processing method comprising:
[0419] generating a management file storing information related to object selection information for selecting object audio data according to a display state of an image; and
[0420] The management file is sent. (19)
[0422] A program that enables a computer to:
[0423] a function of generating a management file storing information related to object selection information for selecting object audio data according to a display state of an image; and
[0424] The function of sending the management file. (20)
[0426] An information processing apparatus includes: a receiving unit configured to receive a management file storing information related to object selection information for selecting object audio data according to a display state of an image. (twenty one)
[0428] The information processing device according to (20), wherein the information related to the object selection information includes the object selection information. (twenty two)
[0430] The information processing device according to (20) or (21), wherein the information related to the object selection information includes metadata file access information for accessing a metadata file including the object selection information. (twenty three)
[0432] The information processing device according to any one of (20) to (22), wherein the management file is a media presentation description (MPD) file. (twenty four)
[0434] The information processing device according to (22), wherein the management file is a Media Presentation Description (MPD) file, and
[0435] The metadata file access information is stored in the AdaptationSet of the MPD file. (25)
[0437] The information processing device according to (24), wherein the metadata file access information is stored in SupplementalProperty of the AdaptationSet. (26)
[0439] The information processing device according to any one of (20) to (25), wherein the object selection information includes direction information regarding a direction in which the image is cut. (27)
[0441] The information processing device according to (26), wherein the direction information includes information on an angle in a horizontal direction and information on an angle in a vertical direction. (28)
[0443] The information processing device according to (27), wherein the direction information also includes information on an angle range relative to a shear direction vector. (29)
[0445] The information processing device according to (26), wherein the direction information includes information on an angle range in a horizontal direction and information on an angle range in a vertical direction. (30)
[0447] The information processing device according to any one of (20) to (29), wherein the object selection information includes viewing angle information related to a viewing angle of the cut image and a viewing angle when viewing relative to the displayed image. (31)
[0449] The information processing device according to (30), wherein the viewing angle information includes information on a horizontal viewing angle range and information on a vertical viewing angle range. (32)
[0451] The information processing device according to (30), wherein the viewing angle information includes information on a magnification range relative to a reference viewing angle. (33)
[0453] The information processing device according to any one of (20) to (26), wherein the object selection information includes position information of two objects between which an angle is to be calculated. (34)
[0455] The information processing device according to (33), wherein the object selection information further includes angle information related to an angle between the two objects. (35)
[0457] The information processing device according to any one of (20) to (34), wherein the object selection information includes information related to the object audio data to be utilized. (36)
[0459] An information processing method to be performed by an information processing apparatus includes receiving a management file storing information related to object selection information for selecting object audio data according to a display state of an image. (37)
[0461] A program that enables a computer to:
[0462] A function of receiving a management file storing information related to object selection information for selecting object audio data according to a display state of an image.
[0463] Reference numerals list
[0464] 100 servers
[0465] 110 Generation Unit
[0466] 111 Data acquisition unit
[0467] 112 Encoding Processing Unit
[0468] 113 Segment file generation unit
[0469] 114 File generation unit
[0470] 120 Control unit
[0471] 130 Communication Unit
[0472] 140 storage units
[0473] 200 Clients
[0474] 210 Processing Units
[0475] 211 File acquisition unit
[0476] 212 File Processing Unit
[0477] 213 Segment file selection unit
[0478] 214 Segment file acquisition unit
[0479] 215 Decoding Processing Unit
[0480] 216 Synthesis Processing Unit
[0481] 220 Control Unit
[0482] 230 Communication Unit
[0483] 240 storage units
[0484] 250 sensor units
[0485] 260 display units
[0486] 270 Speakers
Claims
1. An information processing device, comprising: a generating unit configured to generate a management file storing information related to object selection information for selecting object audio data corresponding to a display state of an image; as well as a sending unit, the sending unit being configured to send the management file, In which, when an object displayed in a first display state of an image is the same as an object displayed in a second display state of the image that is different from the first display state, object audio data corresponding to the first display state is different from object audio data corresponding to the second display state.
2. The information processing device according to claim 1, wherein: The information related to the object selection information includes the object selection information.
3. The information processing device according to claim 1, wherein: The generating unit generates a metadata file including the object selection information, and The information related to the object selection information includes metadata file access information for accessing the generated metadata file.
4. The information processing device according to claim 1, wherein: The management file is a media presentation description MPD file.
5. The information processing device according to claim 3, wherein: The management file is a media presentation description MPD file, and The metadata file access information is stored in the AdaptationSet of the MPD file.
6. The information processing device according to claim 5, wherein: The metadata file access information is stored in the SupplementalProperty of the AdaptationSet.
7. The information processing device according to claim 1, wherein: The object selection information includes direction information regarding a direction in which the image is cut.
8. The information processing device according to claim 7, wherein: The direction information includes information about an angle in a horizontal direction and information about an angle in a vertical direction.
9. The information processing device according to claim 8, wherein: The direction information also includes information about the angle range relative to the shear direction vector.
10. The information processing device according to claim 7, wherein: The direction information includes information about an angle range in the horizontal direction and information about an angle range in the vertical direction.
11. The information processing device according to claim 1, wherein: The object selection information includes viewing angle information related to a viewing angle of the cut image or a viewing angle relative to a display image when viewed.
12. The information processing device according to claim 11, wherein: The viewing angle information includes information about a horizontal viewing angle range and information about a vertical viewing angle range.
13. The information processing device according to claim 11, wherein: The viewing angle information includes information on a magnification range relative to a reference viewing angle.
14. The information processing device according to claim 1, wherein: The object selection information includes position information of two objects between which an angle is to be calculated.
15. The information processing device according to claim 14, wherein: The object selection information further includes angle information regarding an angle between the two objects.
16. The information processing device according to claim 1, wherein: The object selection information includes information related to the object audio data to be utilized.
17. The information processing device according to claim 1, wherein: The generation unit generates an audio file based on a combination of the object audio data corresponding to the display state so that the same object audio data is not included in a plurality of audio files.
18. An information processing method to be performed by an information processing device, the information processing method comprising: generating a management file storing information related to object selection information for selecting object audio data corresponding to a display state of an image; as well as sending said management file, In which, when an object displayed in a first display state of an image is the same as an object displayed in a second display state of the image that is different from the first display state, object audio data corresponding to the first display state is different from object audio data corresponding to the second display state.
19. A program product for causing a computer to implement the following functions: a function of generating a management file storing information related to object selection information for selecting object audio data corresponding to a display state of an image; and The function of sending said management file, in, When an object displayed in a first display state of an image is the same as an object displayed in a second display state of the image different from the first display state, object audio data corresponding to the first display state is different from object audio data corresponding to the second display state.
20. A removable storage medium having a program recorded thereon, the program being used to cause a computer to implement the following functions: a function of generating a management file storing information related to object selection information for selecting object audio data corresponding to a display state of an image; and The function of sending said management file, in, When an object displayed in a first display state of an image is the same as an object displayed in a second display state of the image different from the first display state, object audio data corresponding to the first display state is different from object audio data corresponding to the second display state.
Citation Information
Patent Citations
Information processing device and information processing method
CN106415711A
Encoding device and method, playback device and method, and program
CN106797499A
An apparatus, a method and a computer program for video coding and decoding
WO2017140948A1
