REPRODUCTION DEVICE, REPRODUCTION METHOD, AND PROGRAM

Abstract

To reduce the amount of calculation required for rendering data of a plurality of audio objects. According to one aspect of the present technology, a playback device acquires data of an integrated audio object obtained by integrating audio objects whose sounds cannot be distinguished at a selected assumed listening position from among a plurality of assumed listening positions, and data of other audio objects that have not been integrated, and plays back the data of the integrated audio object and the data of the other audio objects. The present technology can be applied to devices capable of processing object-based audio data.

Description

[Technical field]

[0001] The present technology relates to a playback device, a playback method, and a program, and more particularly to a playback device, a playback method, and a program that enable a reduction in the amount of calculation required for rendering data of a plurality of audio objects. [Background technology]

[0002] Free viewpoint video technology is attracting attention as an approach to video technology. There is a technology that stores an object as a point cloud video by combining images taken from multiple directions by multiple cameras, and generates an image according to the viewing direction and distance (Non-Patent Document 1).

[0003] When viewing of video from any viewpoint becomes a reality, there will also be a demand for audio that can be heard as if one is in a particular location, depending on the viewpoint. For this reason, object-based audio technology has been attracting attention in recent years. Object-based audio data is played back by rendering the waveform data of each audio object into a signal with the desired number of channels that matches the playback system, based on metadata. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] University of Tsukuba website, "HOMETSUKUBA FUTURE-#042: Customize your sports viewing experience with free viewpoint video," [Retrieved March 22, 2017]<URL: http: / / www.tsukuba.ac.jp / notes / 042 / index.html > Summary of the Invention [Problem to be solved by the invention]

[0005] When transmitting object-based audio data, the greater the number of audio objects to be transmitted, the greater the amount of data to be transmitted.

[0006] The present technology has been made in view of these circumstances, and makes it possible to reduce the amount of calculation required for rendering data of multiple audio objects. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present technology includes an acquisition unit that acquires data of an integrated audio object obtained by integrating audio objects whose sounds cannot be distinguished at an assumed listening position selected from a plurality of assumed listening positions, and data of other audio objects that have not been integrated, and a playback unit that plays back the data of the integrated audio object and the data of the other audio objects. The reproduction of the data of the integrated audio object includes a process of reproducing the audio waveform data of each integrated audio object based on the data of the difference components of the audio waveform data of the integrated audio objects.

[0008] In one aspect of the present technology, data of an integrated audio object obtained by integrating audio objects whose sounds cannot be distinguished at a selected assumed listening position from among a plurality of assumed listening positions, and data of other audio objects that have not been integrated, are obtained, and the data of the integrated audio object and the data of the other audio objects are played back. The reproduction of the data of the integrated audio object includes a process of reproducing the audio waveform data of each integrated audio object based on the data of the difference components of the audio waveform data of the integrated audio objects. Effect of the Invention

[0009] According to the present technology, it is possible to reduce the amount of calculation required for rendering data of multiple audio objects.

[0010] Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating a configuration example of a transmission system according to an embodiment of the present technology. [Diagram 2] FIG. 11 is a diagram showing examples of types of objects to be transmitted. [Diagram 3] FIG. 2 is a plan view showing an example of the arrangement of each object. [Figure 4] A diagonal view of the venue. [Diagram 5] FIG. 4 is a front view showing an example of the arrangement of each object. [Figure 6] FIG. 2 is a plan view showing an example of the arrangement of each object. [Figure 7] FIG. 2 is a plan view showing an example of the arrangement of each object including an integrated object. [Figure 8] FIG. 2 is a front view showing an example of the arrangement of objects including an integrated object. [Figure 9] FIG. 2 is a block diagram showing a configuration example of a content generating device. [Figure 10] 2 is a block diagram showing an example of a functional configuration of a content generating device. FIG. [Figure 11] FIG. 2 is a block diagram showing an example of a functional configuration of a playback device. [Figure 12] 11 is a flowchart illustrating a content generation process of the content generation device. [Figure 13] 13 is a flowchart illustrating an integration process of the content generating device. [Figure 14] 11 is a flowchart illustrating a transmission process of the content generating device. [Figure 15] 11 is a flowchart illustrating a playback process of the playback device. [Figure 16] FIG. 13 is a diagram showing another example of the arrangement of objects. [Figure 17] FIG. 13 is a diagram showing another example of how objects are grouped together. [Figure 18] FIG. 13 is a diagram showing yet another example of how objects are grouped together. [Figure 19] FIG. 11 is a diagram illustrating an example of transmission of flag information. [Figure 20] 13A and 13B are diagrams illustrating another example of transmission of flag information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Transmission System Configuration 2. How to organize objects 3. Example of each device configuration 4. Operation of each device 5. Variations in how objects are grouped 6. Variations

[0013] <<Transmission system configuration>> FIG. 1 is a diagram illustrating an example of a configuration of a transmission system according to an embodiment of the present technology.

[0014] The transmission system in FIG. 1 is configured by connecting a content generating device 1 and a playback device 2 via the Internet 3.

[0015] The content generation device 1 is a device managed by a content creator, and is installed in a venue #1 where a live music concert is being held. The content generated by the content generation device 1 is transmitted to a playback device 2 via the Internet 3. The content may be distributed via a server (not shown).

[0016] On the other hand, the playback device 2 is a device installed in the home of a user who wants to view the live music content generated by the content generation device 1. In the example of Fig. 1, only the playback device 2 is shown as a playback device that receives distributed content, but in reality, many playback devices are connected to the Internet 3.

[0017] The video of the content generated by the content generation device 1 is a video that allows the viewpoint to be switched. The audio of the content is also a sound that allows the viewpoint (assumed listening position) to be switched so that the listening position is the same as the viewpoint position of the video, for example. When the viewpoint is switched, the localization of the sound is switched.

[0018] The audio of the content is prepared as object-based audio. The audio data included in the content includes audio waveform data of each audio object and rendering parameters as metadata for localizing the sound source of each audio object. Hereinafter, audio objects will be simply referred to as objects.

[0019] A user of the playback device 2 can select an arbitrary viewpoint from a plurality of prepared viewpoints, and view the content with video and audio according to the viewpoint.

[0020] Content including video data of an image viewed from a viewpoint selected by the user and object-based audio data of the viewpoint selected by the user is provided from the content generating device 1 to the playback device 2. For example, such object-based audio data is transmitted in a form compressed by a predetermined format such as MPEG-H 3D Audio.

[0021] Details of MPEG-H 3D Audio are disclosed in ISO / IEC 23008-3:2015 “Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 3: 3D audio”, < https: / / www.iso.org / standard / 63878.html>.

[0022] The following mainly describes the processing related to audio data. As shown in Fig. 1, a live music performance taking place at venue #1 is a live performance on stage with five people playing bass, drums, guitar 1 (main guitar), guitar 2 (side guitar), and vocals. The bass, drums, guitar 1, guitar 2, and vocals are treated as objects, and audio waveform data for each object and rendering parameters for each viewpoint are generated in the content generation device 1.

[0023] FIG. 2 is a diagram showing examples of types of objects transmitted from the content generator 1. As shown in FIG.

[0024] For example, when viewpoint 1 is selected by the user from among multiple viewpoints, data of five types of objects, bass, drums, guitar 1, guitar 2, and vocals, is transmitted as shown in A of Fig. 2. The transmitted data includes audio waveform data of each of the objects, bass, drums, guitar 1, guitar 2, and vocals, and rendering parameters of each object for viewpoint 1.

[0025] Also, when viewpoint 2 is selected by the user, Guitar 1 and Guitar 2 are grouped together as a single object, Guitar, and data for four types of objects, bass, drums, guitar, and vocals, is transmitted, as shown in B of Fig. 2. The transmitted data includes audio waveform data for each of the bass, drums, guitar, and vocals objects, and rendering parameters for each object for viewpoint 2.

[0026] Viewpoint 2 is set at a position where, for example, the human ear cannot distinguish between the sound of guitar 1 and the sound of guitar 2 because they are heard from the same direction. In this way, objects that cannot be distinguished from the viewpoint selected by the user are grouped together as a single object and data is transmitted.

[0027] By appropriately grouping objects together and transmitting data according to the selected viewpoint, it becomes possible to reduce the amount of data transmitted.

[0028] <<How to group objects>> Here, a method for grouping objects will be described.

[0029] (1) Assume there are multiple objects. The audio waveform data for an object is defined as follows: x(n,i) i=0,1,2,…,L-1

[0030] n is the time index, and i represents the type of object, where L is the number of objects.

[0031] (2) Assume there are multiple perspectives. The rendering information of an object corresponding to each viewpoint is defined as follows: r(i,j) j=0,1,2,…,M-1

[0032] j represents the type of viewpoint. The number of viewpoints is M.

[0033] (3) Audio data y(n,j) corresponding to each viewpoint is expressed by the following equation (1).

number

[0034] Here, it is assumed that the rendering information r is a gain (gain information). In this case, the value range of the rendering information r is 0 to 1. The audio data of each viewpoint is expressed as the result of multiplying the audio waveform data of each object by a gain and adding up the audio waveform data of all objects. The calculation shown in equation (1) is performed in the playback device 2.

[0035] (4) A plurality of objects whose sounds cannot be distinguished from the viewpoint are transmitted together. For example, an object that is far from the viewpoint and whose horizontal angle from the viewpoint is within a predetermined range is selected as an object whose sounds cannot be distinguished. On the other hand, an object that is close and whose sounds can be distinguished from the viewpoint is transmitted as an independent object without being grouped together.

[0036] (5) The rendering information of an object corresponding to each viewpoint is defined as follows according to the type of object, the position of the object, and the position of the viewpoint: r(obj_type, obj_loc_x, obj_loc_y, obj_loc_z, lis_loc_x, lis_loc_y, lis_loc_z)

[0037] The obj_type is information indicating the type of object, for example, the type of musical instrument.

[0038] obj_loc_x, obj_loc_y, obj_loc_z are information indicating the position of an object in three-dimensional space.

[0039] lis_loc_x, lis_loc_y, lis_loc_z are information indicating the position of the viewpoint in three-dimensional space.

[0040] For an object to be transmitted independently, parameter information composed of obj_type, obj_loc_x, obj_loc_y, obj_loc_z, lis_loc_x, lis_loc_y, and lis_loc_z is transmitted together with the rendering information r. The rendering parameters are composed of parameter information and rendering information.

[0041] The specific details will be explained below.

[0042] (6) For example, let us say that the objects of bass, drums, guitar 1, guitar 2, and vocals are arranged as shown in Fig. 3. Fig. 3 is a top view of stage #11 in venue #1.

[0043] (7) For venue #1, the XYZ axes are set as shown in Figure 4. Figure 4 is a view of venue #1, including stage #11 and the spectator seats, seen from an oblique angle. Origin O is the center position on stage #11. Viewpoints 1 and 2 are set in the spectator seats.

[0044] The coordinates of each object are expressed as follows, in meters: Base coordinates: x=-20, y=0, z=0 Drum coordinates: x=0, y=-10, z=0 Coordinates of guitar 1: x=20, y=0, z=0 Coordinates of guitar 2: x=30, y=0, z=0 Vocal coordinates: x=0, y=10, z=0

[0045] (8) The coordinates of each viewpoint are expressed as follows: Viewpoint 1: x=25, y=30, z=-1 Viewpoint 2: x=-35, y=30, z=-1

[0046] It should be noted that the positions of each object and each viewpoint in the diagram merely represent an image of the positional relationships, and are not positions that accurately reflect the above numerical values.

[0047] (9) At this time, the rendering information for each object at viewpoint 1 is expressed as follows: Base Rendering Information :r(0, -20, 0, 0, 25, 30, -1) Drum Rendering Information :r(1, 0, -10, 0, 25, 30, -1) Guitar 1 rendering info :r(2, 20, 0, 0, 25, 30, -1) Guitar 2 rendering info :r(3, 30, 0, 0, 25, 30, -1) Vocal Rendering Information :r(4, 0, 10, 0, 25, 30, -1)

[0048] The obj_type of each object shall take the following values. Base :obj_type=0 Drum :obj_type=1 Guitar 1: obj_type=2 Guitar 2: obj_type=3 Vocal: obj_type=4

[0049] For viewpoint 2 as well, the content generator 1 generates rendering parameters including the parameter information and rendering information expressed as above.

[0050] (10) From the above equation (1), the audio data when viewpoint 1 (j=0) is selected is expressed as in the following equation (2).

number

[0051] where x(n,i) represents the following object: i=0: Base object i=1: Drum object i=2: Guitar 1 object i=3: Guitar 2 object i=4: Vocal object

[0052] An example of the arrangement of each object as viewed from viewpoint 1 is shown in A of Fig. 5. In A of Fig. 5, the lower part shown in a lighter color indicates the side of stage #11. The same is true for the other figures.

[0053] (11) Similarly, the audio data when viewpoint 2 (j=1) is selected is expressed by the following equation (3).

number

[0054] An example of the arrangement of each object as seen from viewpoint 2 is shown in FIG.

[0055] (12) Here, as shown in Figure 6, angle θ1, which is the horizontal angle between the directions of guitar 1 and guitar 2 when viewpoint 1 is used as the reference, is different from angle θ2, which is the horizontal angle between the directions of guitar 1 and guitar 2 when viewpoint 2 is used as the reference. Angle θ2 is narrower than angle θ1.

[0056] 6 is a plan view showing the positional relationship between each object and the viewpoint. Angle θ1 is the angle between dashed line A1-1 connecting viewpoint 1 and guitar 1, and dashed line A1-2 connecting viewpoint 1 and guitar 2. Moreover, angle θ2 is the angle between dashed line A2-1 connecting viewpoint 2 and guitar 1, and dashed line A2-2 connecting viewpoint 2 and guitar 2.

[0057] (13) The angle θ1 is an angle that is discriminable to the human hearing, that is, the sound of guitar 1 and the sound of guitar 2 can be identified as sounds coming from different directions. On the other hand, the angle θ2 is an angle that is indistinguishable to the human hearing. In this case, the audio data of viewpoint 2 can be replaced as shown in the following formula (4).

number

[0058] In equation (4), x(n,5) is expressed by the following equation (5).

number

[0059] That is, equation (5) combines guitar 1 and guitar 2 into a single object, and expresses the audio waveform data of this single object as the sum of the audio waveform data of guitar 1 and the audio waveform data of guitar 2. The obj_type of the integrated object, which is a single object combining guitar 1 and guitar 2, is obj_type=5.

[0060] The rendering information of the integrated object is expressed as an average of the rendering information of guitar 1 and the rendering information of guitar 2, for example, by the following formula (6).

number

[0061] In this way, for an integrated object represented as obj_type=5, processing is performed with audio waveform data set as x(n,5) and rendering information set as r(5, 25, 0, 0, -35, 30, -1). Figure 7 shows an example of the placement of each object when Guitar 1 and Guitar 2 are combined into one object.

[0062] Fig. 8 shows an example of the placement of each object, including the integrated object, as seen from viewpoint 2. Guitar 1 and Guitar 2 are shown in the image from viewpoint 2, but only one guitar is placed as an audio object.

[0063] (14) In this way, objects that are auditorily indistinguishable from the selected viewpoint are grouped together and data is transmitted as one object.

[0064] This allows the content generator 1 to reduce the number of objects for which data is transmitted, thereby reducing the amount of data transmitted. In addition, since the number of objects to be rendered is small, the playback device 2 can reduce the amount of calculation required for rendering.

[0065] In the example of Figure 6, in addition to guitar 1 and guitar 2, vocals are also objects that are within the range of horizontal angle θ2 when viewed from viewpoint 2. However, the vocals are closer to viewpoint 2 and are therefore distinguishable from guitar 1 and guitar 2.

[0066] <<Examples of device configuration>> <Configuration of Content Generation Device 1> FIG. 9 is a block diagram showing an example of the configuration of the content generator 1. As shown in FIG.

[0067] A CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, and a RAM (Random Access Memory) 23 are interconnected by a bus 24. An input / output interface 25 is further connected to the bus 24. An input unit 26, an output unit 27, a storage unit 28, a communication unit 29, and a drive 30 are connected to the input / output interface 25.

[0068] The input unit 26 is composed of a keyboard, a mouse, etc. The input unit 26 outputs a signal representing the content of a user's operation.

[0069] The output unit 27 is composed of a display such as an LCD (Liquid Crystal Display) or an organic EL display, and a speaker.

[0070] The storage unit 28 is configured with a hard disk, a non-volatile memory, etc. The storage unit 28 stores various data such as programs executed by the CPU 21 and contents.

[0071] The communication unit 29 is configured with a network interface and the like, and communicates with external devices via the Internet 3 .

[0072] The drive 30 writes data to the attached removable medium 31 and reads data recorded on the removable medium 31 .

[0073] The playback device 2 has the same configuration as that shown in Fig. 9. In the following description, the configuration shown in Fig. 9 will be cited as the configuration of the playback device 2 where appropriate.

[0074] FIG. 10 is a block diagram showing an example of a functional configuration of the content generator 1. As shown in FIG.

[0075] At least a part of the configuration shown in Fig. 10 is realized by executing a predetermined program by CPU 21 in Fig. 9. In content generation device 1, an audio encoder 51, a metadata encoder 52, an audio generation unit 53, a video generation unit 54, a content storage unit 55, and a transmission control unit 56 are realized.

[0076] The audio encoder 51 acquires an audio signal during a live music performance collected by a microphone (not shown), and generates audio waveform data for each object.

[0077] The metadata encoder 52 generates rendering parameters for each object for each viewpoint in accordance with the operation by the content creator. The rendering parameters for each of the multiple viewpoints set in venue #1 are generated by the metadata encoder 52.

[0078] The audio generation unit 53 generates object-based audio data for each viewpoint by associating the audio waveform data generated by the audio encoder 51 with the rendering parameters generated by the metadata encoder 52. The audio generation unit 53 outputs the generated audio data for each viewpoint to the content storage unit 55.

[0079] An integration unit 61 is implemented in the audio generation unit 53. The integration unit 61 appropriately integrates objects. For example, the integration unit 61 reads out audio data of each viewpoint stored in the content storage unit 55, integrates objects that can be integrated, and stores the integrated audio data in the content storage unit 55.

[0080] The video generation unit 54 acquires video data captured by cameras installed at the positions of the respective viewpoints, and generates video data for each viewpoint by encoding the data using a predetermined encoding method. The video generation unit 54 outputs the generated video data for each viewpoint to the content storage unit 55.

[0081] The content storage unit 55 stores the audio data for each viewpoint generated by the audio generation unit 53 and the video data for each viewpoint generated by the video generation unit 54 in association with each other.

[0082] The transmission control unit 56 controls the communication unit 29 to communicate with the playback device 2. The transmission control unit 56 receives selected viewpoint information, which is information indicating a viewpoint selected by a user of the playback device 2, and transmits to the playback device 2 content consisting of video data and audio data corresponding to the selected viewpoint.

[0083] <Configuration of playback device 2> FIG. 11 is a block diagram showing an example of the functional configuration of the playback device 2.

[0084] At least a part of the configuration shown in Fig. 11 is realized by executing a predetermined program by CPU 21 in Fig. 9. In the playback device 2, a content acquisition unit 71, a separation unit 72, an audio playback unit 73, and a video playback unit 74 are realized.

[0085] When a viewpoint is selected by the user, the content acquisition unit 71 controls the communication unit 29 to transmit selected viewpoint information to the content generation device 1. The content acquisition unit 71 receives and acquires content transmitted from the content generation device 1 in response to the transmission of the selected viewpoint information. Content including video data and audio data corresponding to the viewpoint selected by the user is transmitted from the content generation device 1. The content acquisition unit 71 outputs the acquired content to the separation unit 72.

[0086] The separator 72 separates the video data and the audio data contained in the content supplied from the content acquisition unit 71. The separator 72 outputs the video data of the content to a video playback unit 74, and outputs the audio data to an audio playback unit 73.

[0087] The audio playback unit 73 renders the audio waveform data constituting the audio data supplied from the separation unit 72 based on the rendering parameters, and outputs the sound of the content from the speaker constituting the output unit 27 .

[0088] The video playback unit 74 decodes the video data supplied from the separation unit 72 and displays an image of the content from a predetermined viewpoint on a display constituting the output unit 27 .

[0089] The speaker and display used for playing back the content may be provided as external devices connected to the playback device 2.

[0090] <<Operation of each device>> Next, the operations of the content generation device 1 and playback device 2 having the above-mentioned configuration will be described.

[0091] <Operation of Content Generation Device 1> Content generation processing First, the process of the content generating device 1 for generating content will be described with reference to the flowchart of FIG.

[0092] The process of FIG. 12 starts, for example, when a live music performance starts and video data from each viewpoint and audio signals from each object are input to the contents generator 1.

[0093] Multiple cameras are installed in venue #1, and images captured by these cameras are input to content generation device 1. In addition, microphones are installed near each object in venue #1, and audio signals picked up by these microphones are input to content generation device 1.

[0094] In step S1, the video generation unit 54 acquires video data captured by cameras for each viewpoint, and generates video data for each viewpoint.

[0095] In step S2, the audio encoder 51 acquires the audio signal of each object and generates audio waveform data for each object. In the above example, audio waveform data is generated for each of the objects: bass, drums, guitar 1, guitar 2, and vocals.

[0096] In step S3, the metadata encoder 52 generates rendering parameters for each object for each viewpoint as manipulated by the content creator.

[0097] For example, if viewpoint 1 and viewpoint 2 are set to venue #1 as described above, a set of rendering parameters for each of the objects (bass, drums, guitar 1, guitar 2, and vocals) at viewpoint 1, and a set of rendering parameters for each of the objects (bass, drums, guitar 1, guitar 2, and vocals) at viewpoint 2 are generated.

[0098] In step S4, the content storage unit 55 generates and stores content for each viewpoint by associating the audio data and the video data with each viewpoint.

[0099] The above process is repeated while the live music performance is being performed. For example, when the live music performance has ended, the process of FIG.

[0100] Object integration processing Next, the process of the contents generator 1 for integrating objects will be described with reference to the flowchart of FIG.

[0101] For example, the process of FIG. 13 is performed at a predetermined timing after the audio waveform data of each of the bass, drum, guitar 1, guitar 2, and vocal objects and a set of rendering parameters for each object at each viewpoint are generated.

[0102] In step S11, the integration unit 61 focuses on one predetermined viewpoint among the multiple viewpoints for which rendering parameters have been generated.

[0103] In step S12, the integration unit 61 identifies the position of each object based on the parameter information included in the rendering parameters, and obtains the distance to each object based on the viewpoint of interest.

[0104] In step S13, the integration unit 61 judges whether or not there are multiple objects that are far from the viewpoint of interest. For example, an object that is located at a distance greater than a preset threshold is treated as a distant object. If it is judged in step S13 that there are no multiple distant objects, the process returns to step S11, and the viewpoint of interest is switched and the above process is repeated.

[0105] On the other hand, if it is determined in step S13 that there are multiple distant objects, the process proceeds to step S14. If viewpoint 2 is selected as the viewpoint of interest, for example, drums, guitar 1, and guitar 2 are determined to be distant objects.

[0106] In step S14, the integration unit 61 determines whether or not a plurality of distant objects are within a predetermined horizontal angle range. That is, in this example, an object that is far away from the viewpoint and has a horizontal angle within a predetermined angle range as viewed from the viewpoint is processed as an object whose sound cannot be discriminated.

[0107] If it is determined in step S14 that the multiple distant objects are not within the range of the predetermined horizontal angle, the integration unit 61 sets all objects for the viewpoint of interest as transmission targets in step S15. In this case, when the viewpoint of interest is selected during content transmission, audio waveform data of all objects and rendering parameters of each object for that viewpoint are transmitted in the same manner as when viewpoint 1 described above is selected.

[0108] On the other hand, if it is determined in step S14 that the multiple distant objects are within the range of the predetermined horizontal angle, the integration unit 61 integrates the multiple distant objects within the range of the predetermined horizontal angle in step S16, and sets the integrated object as a transmission target. In this case, when the viewpoint of interest is selected during content transmission, the audio waveform data and rendering parameters of the integrated object are transmitted together with the audio waveform data and rendering parameters of the independent objects that are not integrated.

[0109] In step S17, the integration unit 61 generates audio waveform data of the integrated object by calculating the sum of the audio waveform data of objects that are far away and within a range of a predetermined horizontal angle. This process corresponds to the process of calculating the above formula (5).

[0110] In step S18, the integration unit 61 generates rendering parameters of the integrated object by averaging the rendering parameters of objects that are far away and within a range of a predetermined horizontal angle. This process corresponds to the process of calculating the above formula (6).

[0111] The audio waveform data and rendering parameters of the integrated object are stored in the content storage unit 55 and are managed as data to be transmitted when a viewpoint of interest is selected.

[0112] After the transmission target is set in step S15, or after the rendering parameters of the integrated object are generated in step S18, the integration unit 61 determines whether or not all viewpoints have been focused on in step S19. If it is determined in step S19 that there is a viewpoint that has not been focused on, the process returns to step S11, and the viewpoint to be focused on is switched and the above processing is repeated.

[0113] On the other hand, if it is determined in step S19 that attention has been paid to all the viewpoints, the processing in FIG. 13 ends.

[0114] Through the above process, objects whose sounds cannot be differentiated from one another from a certain viewpoint are grouped together as an integrated object.

[0115] The processing in Fig. 13 may be performed in response to selected viewpoint information being transmitted from the playback device 2. In this case, the processing in Fig. 13 is performed with attention paid to the viewpoint selected by the user, and objects are integrated as appropriate.

[0116] Instead of objects that are far from the viewpoint and whose horizontal angle from the viewpoint is within a predetermined angle range, objects that are simply far from the viewpoint may be processed as objects whose sounds cannot be discriminated. Also, objects whose horizontal angle from the viewpoint is within a predetermined angle range may be processed as objects whose sounds cannot be discriminated.

[0117] Distances between objects may be calculated and objects that are closer than a threshold distance may be grouped together as an integrated object.

[0118] When the amount of components that mask the audio waveform data of one object is greater than a threshold, the objects may be processed as objects whose sounds cannot be distinguished. In this way, the method of determining an object whose sounds cannot be distinguished is arbitrary.

[0119] Content transmission processing Next, the process of the content generator 1 for transmitting content will be described with reference to the flowchart of FIG.

[0120] For example, the process in FIG. 14 is started when a request to start content transmission is made by the playback device 2 and selected viewpoint information is transmitted from the playback device 2.

[0121] In step S31, the transmission control unit 56 receives the selected viewpoint information transmitted from the playback device 2.

[0122] In step S32, the transmission control unit 56 reads and transmits the video data of the viewpoint selected by the user of the playback device 2, and the audio waveform data and rendering parameters of each object at the selected viewpoint from the content storage unit 55. For an integrated object, the audio waveform data and rendering parameters generated as the audio data of the integrated object are transmitted.

[0123] The above process is repeated until the transmission of the content is completed, at which point the process of FIG.

[0124] <Operation of playback device 2> Next, the process of the playback device 2 playing back content will be described with reference to the flowchart of FIG.

[0125] In step S101, the content acquisition unit 71 transmits information representing a viewpoint selected by a user to the content generation device 1 as selected viewpoint information.

[0126] For example, before starting to view content, a screen used to select which of a plurality of prepared viewpoints to view the content from is displayed based on information transmitted from the content generation device 1. In response to transmitting the selected viewpoint information, the content generation device 1 transmits content including video data and audio data of the viewpoint selected by the user.

[0127] In step S102, the content acquisition unit 71 receives and acquires the content transmitted from the content generator 1.

[0128] In step S103, the separator 72 separates the video data and the audio data contained in the content.

[0129] In step S104, the video playback unit 74 decodes the video data supplied from the separation unit 72, and displays an image of the content from a predetermined viewpoint on the display.

[0130] In step S105, the audio playback unit 73 renders the audio waveform data of each object included in the audio data supplied from the separation unit 72 based on the rendering parameters of each object, and outputs the sound from the speaker.

[0131] The above process is repeated until the playback of the content is completed, at which point the process of FIG.

[0132] By carrying out the above-described series of processes, it is possible to reduce the number of objects to be transmitted, and therefore the amount of data to be transmitted.

[0133] <<Variations in how objects are grouped>> (1) Aggregation according to transmission bit rate The maximum number of objects may be determined according to the transmission bit rate, and objects may be grouped together so as not to exceed this limit.

[0134] Fig. 16 is a diagram showing another example of object arrangement. Fig. 16 shows an example of performance by bass, drums, guitar 1, guitar 2, vocals 1 to 6, piano, trumpet, and saxophone. In the example of Fig. 16, viewpoint 3 is set to view stage #11 from the front.

[0135] For example, if the maximum number of objects according to the transmission bit rate is 3 and viewpoint 3 is selected, piano, bass, vocal 1, and vocal 2 are grouped together as the first object based on the angle determination described above. Piano, bass, vocal 1, and vocal 2 are objects within the angle range between dashed lines A11 and A12, which are set toward the left of stage #11 with viewpoint 3 as the base.

[0136] Similarly, the drums, vocals 3, and vocals 4 are grouped together as a second object. The drums, vocals 3, and vocals 4 are objects within the range of the angle between the dashed lines A12 and A13, which are set toward the center of the stage #11.

[0137] Additionally, the trumpet, saxophone, guitar 1, guitar 2, vocalist 5, and vocalist 6 are grouped together as a third object. The trumpet, saxophone, guitar 1, guitar 2, vocalist 5, and vocalist 6 are objects within the range of the angle between dashed lines A13 and A14, which are set toward the right of stage #11.

[0138] As described above, audio waveform data and rendering parameters for each object (integrated object) are generated, and audio data for the three objects is transmitted. In this way, it is possible to group three or more objects into an integrated object.

[0139] Fig. 17 is a diagram showing another example of how to group objects. For example, when the maximum number of objects according to the transmission bit rate is 6 and viewpoint 3 is selected, the objects are grouped as shown by the broken lines in Fig. 17 based on the determination based on the angle and distance as described above.

[0140] In the example of Fig. 17, the piano and bass are grouped together as the first object, and vocal 1 and vocal 2 are grouped together as the second object. The drums are an independent third object, and vocal 3 and vocal are grouped together as the fourth object. The trumpet, saxophone, guitar 1, and guitar 2 are grouped together as the fifth object, and vocal 5 and vocal 6 are grouped together as the sixth object.

[0141] The grouping method shown in FIG. 16 is the grouping method selected when the transmission bit rate is lower than that shown in FIG.

[0142] By determining the number of objects to be transmitted according to the transmission bit rate, content can be transmitted with sound quality according to the transmission bit rate, such that high sound quality is possible when the transmission bit rate is high and low sound quality is possible when the transmission bit rate is low.

[0143] For example, in the content storage unit 55 of the content generation device 1, audio data for three objects as shown in FIG. 16 and audio data for six objects as shown in FIG. 17 are stored as audio data to be transmitted when viewpoint 3 is selected.

[0144] Before starting the transmission of content, the transmission control unit 56 determines the communication environment of the playback device 2 and, depending on the transmission bit rate, selects and transmits either the audio data of three objects or the audio data of six objects.

[0145] (2) Grouping objects In the above examples, the rendering information is assumed to be gain, but it is also possible to use reverb information. Among the parameters that make up reverb information, the most important parameter is the amount of reverberation. The amount of reverberation is the amount of spatial reflection components from walls, floors, etc. The amount of reverberation differs depending on the distance between the object (musical instrument) and the listener. Generally, the shorter the distance, the less the amount of reverberation, and the longer the distance, the more the amount of reverberation.

[0146] In addition to judging whether sounds are distinguishable based on distance and angle and grouping objects, objects may be grouped according to the distance between objects as another index. An example of grouping objects taking into account the distance between objects is shown in FIG. 18.

[0147] In the example of Figure 18, the objects are divided into groups as shown by the broken lines, and the objects belonging to each group are collected. The objects belonging to each group are as follows: Group 1 Vocal 1, Vocal 2 Group 2 Vocal 3, Vocal 4 Group 3 Vocal 5, Vocal 6 Group 4 Bass Group 5 Piano Group 6 Drums Group 7 Guitar 1, 2 Group 8 Trumpet, Saxophone

[0148] In this case, the content storage unit 55 of the content generator 1 stores audio data of eight objects as audio data to be transmitted when viewpoint 3 is selected.

[0149] In this way, objects within an angular range in which sounds cannot be distinguished may still be treated as objects to which different reverbs are applied.

[0150] In this way, groups of objects that can be grouped together can be predefined, and only objects that belong to the same group and satisfy the above-mentioned conditions based on distance and angle will be grouped together into an integrated object.

[0151] Groups may be set according to the type of object, the position of the object, and the like, in addition to the distance between the objects.

[0152] In addition to the gain and reverb information, the rendering information may be equalizer information, compressor information, or reverb information. That is, the rendering information r can be information that represents at least one of the gain, equalizer information, compressor information, and reverb information.

[0153] (3) High-efficiency object audio coding The case where two stringed instrument objects are combined into one stringed instrument object will be described. A new object type (obj_type) is assigned to the one stringed instrument object as an integrated object.

[0154] If the audio waveform data of the object to be consolidated, violin 1, is x(n,10) and the audio waveform data of violin 2 is x(n,11), the audio waveform data x(n,14) of the string instrument object as the integrated object is expressed by the following equation (7).

number

[0155] Here, since violin 1 and violin 2 are the same string instrument, the correlation between the two audio waveform data is high.

[0156] The difference component x(n,15) of the audio waveform data of violin 1 and violin 2 shown in the following equation (8) has low information entropy and requires a small bit rate when encoded.

number

[0157] By transmitting the difference component x(n,15) shown in equation (8) together with the audio waveform data x(n,14) represented as the sum component, it is possible to achieve high sound quality at a low bit rate, as will be explained below.

[0158] It is assumed that audio waveform data x(n,14) is normally transmitted from the content generation device 1 to the playback device 2. Here, when sound quality is improved on the playback device 2 side, the difference component x(n,15) is also transmitted.

[0159] The playback device 2 receives the difference component x(n,15) along with the audio waveform data x(n,14), and can reproduce the audio waveform data x(n,10) of violin 1 and the audio waveform data x(n,11) of violin 2 by performing the calculations shown in the following equations (9) and (10).

number

number

[0160] In this case, the content storage unit 55 of the content generation device 1 stores the difference component x(n,15) together with the audio waveform data x(n,14) as the audio data of the string instrument object to be transmitted when a specified viewpoint is selected.

[0161] A flag indicating that the differential component data is held is managed in the content generation device 1. The flag is transmitted from the content generation device 1 to the playback device 2 together with, for example, other information, and the playback device 2 identifies that the differential component data is held.

[0162] In this way, for audio waveform data of highly correlated objects, by storing the difference component on the content generating device 1 side, it becomes possible to adjust the sound quality according to the transmission bit rate in two stages. That is, when the communication environment of the playback device 2 is good (when the transmission bit rate is high), the audio waveform data x(n,14) and the difference component x(n,15) are transmitted, and when the communication environment is not good, only the audio waveform data x(n,14) is transmitted.

[0163] The amount of data obtained by adding together the audio waveform data x(n,14) and the difference component x(n,15) is less than the amount of data obtained by adding together the audio waveform data x(n,10) and x(n,11).

[0164] When the number of objects is four, they can be grouped in the same manner. When the four instruments are grouped, the audio waveform data x(n,14) of the grouped objects is expressed by the following formula (11).

number

[0165] Here, x(n,10) is the audio waveform data of violin 1, x(n,11) is the audio waveform data of violin 2, x(n,12) is the audio waveform data of violin 3, and x(n,13) is the audio waveform data of violin 4.

[0166] In this case, the content generating device 1 holds the data of the difference components expressed by the following equations (12) to (14).

number

number

number

[0167] Normally, audio waveform data x(n,14) is transmitted from the content generating device 1 to the playback device 2. Here, when the playback device 2 performs high-quality sound enhancement, difference components x(n,15), x(n,16), and x(n,17) are also transmitted.

[0168] The playback device 2 receives the difference components x(n,15), x(n,16), and x(n,17) along with the audio waveform data x(n,14), and can reproduce the audio waveform data x(n,10) of violin 1, audio waveform data x(n,11) of violin 2, audio waveform data x(n,12) of violin 3, and audio waveform data x(n,13) of violin 4 by performing the calculations shown in the following equations (15) to (18).

number

number

number

number

[0169] Furthermore, from the following equation (19), it can be seen that with the audio waveform data x(n,14) and the difference component x(n,15), it is possible to obtain the sum (x(n,10) + x(n,11)) of the audio waveform data of Violin 1 and the audio waveform data of Violin 2. Furthermore, from the following equation (20), it can be seen that with the audio waveform data x(n,14) and the difference component x(n,15), it is possible to obtain the sum (x(n,12) + x(n,13)) of the audio waveform data of Violin 3 and the audio waveform data of Violin 4.

number

number

[0170] For example, when the transmission bit rate that the playback device 2 can handle is higher than the first threshold and the communication environment is the best of the three stages, the difference components x(n,15), x(n,16), and x(n,17) are transmitted from the content generation device 1 along with the audio waveform data x(n,14) that summarizes the four objects.

[0171] In the playback device 2, the calculations shown in equations (15) to (18) are performed, audio waveform data of each object of violin 1, violin 2, violin 3, and violin 4 is obtained, and high quality playback is performed.

[0172] Furthermore, when the transmission bit rate that the playback device 2 can handle is less than the first threshold but higher than the second threshold, and the communication environment is relatively good, the difference component x(n,15) is transmitted from the content generation device 1 together with the audio waveform data x(n,14) that summarizes the four objects.

[0173] In playback device 2, the calculations shown in equations (19) and (20) are performed, and audio waveform data combining violin 1 and violin 2, and audio waveform data combining violin 3 and violin 4 are obtained, thereby performing playback with higher quality than when only the audio waveform data x(n,14) is used.

[0174] When the transmission bit rate that the playback device 2 can handle is less than the second threshold, the content generator 1 transmits audio waveform data x(n,14) that compiles the four objects.

[0175] In this way, the content generator 1 may perform hierarchical transmission (encoding) according to the transmission bit rate.

[0176] Such hierarchical transmission may be performed according to the fee paid by the user of the playback device 2. For example, if the user pays a normal fee, only the audio waveform data x(n,14) is transmitted, and if the user pays a higher fee, the audio waveform data x(n,14) and the difference component are transmitted.

[0177] (4) Linking with point cloud video data It is assumed that the video data of the content transmitted by the content generation device 1 is point cloud moving image data. Both the point cloud moving image data and the object audio data have coordinate data in three-dimensional space, and become color data and audio data at those coordinates.

[0178] For point cloud video data, see, for example, Microsoft “A Voxelized Point Cloud Dataset”,<https: / / jpeg.org / plenodb / pc / microsoft / > " is disclosed.

[0179] The content generation device 1 holds, for example, three-dimensional coordinates as position information of vocals, and holds the point cloud video data and audio object data linked to the coordinates, which allows the playback device 2 to easily obtain the point cloud video data and audio object data of a desired object.

[0180] <<Modifications>> An audio bitstream transmitted by the content generator 1 may contain flag information indicating whether the objects transmitted by the stream are unconsolidated independent objects or integrated objects. An audio bitstream containing flag information is shown in FIG.

[0181] The audio bitstream in FIG. 19 also includes, for example, audio waveform data and rendering parameters for the objects.

[0182] The flag information in FIG. 19 may be information indicating whether an object transmitted by a stream is an independent object or whether it is an integrated object or not.

[0183] This enables the playback device 2 to analyze the stream and identify whether the data contained in the stream is data of an integrated object or data of an independent object.

[0184] Such flag information may be described in a playback management file transmitted together with a bitstream, as shown in Fig. 20. The playback management file also describes information such as the stream ID of a stream to be played back by the playback management file (a stream played back using the playback management file). This playback management file may be configured as an MPD (Media Presentation Description) file of MPEG-DASH.

[0185] This allows the playback device 2 to refer to the playback management file and identify whether the object transmitted by the stream is an integrated object or an independent object.

[0186] Although the content played back by the playback device 2 has been described as including video data and object-based audio data, the content may be made up of object-based audio data without including video data. When a specific listening position is selected from listening positions for which rendering parameters are prepared, each audio object is played back using the rendering parameters for the selected listening position.

[0187] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0188] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0189] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0190] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0191] The effects described herein are merely examples and are not limiting, and other effects may also be present.

[0192] About the Program The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0193] The program to be installed is provided by being recorded on removable medium 31 shown in FIG. 9, which is made up of an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 22 or storage unit 28.

[0194] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0195] About combinations The present technology can also be configured as follows. (1) an integration unit that integrates, among a plurality of audio objects for a predetermined assumed listening position among a plurality of assumed listening positions, audio objects whose sounds cannot be discriminated at the predetermined assumed listening position; a transmission unit that transmits data of the integrated audio object obtained by the integration together with data of other audio objects whose sounds can be discriminated at the predetermined assumed listening position; An information processing device comprising: (2) The integration unit generates audio waveform data and rendering parameters of the integrated audio object based on audio waveform data and rendering parameters of the plurality of audio objects to be integrated. The information processing device according to (1). (3) The transmission unit transmits, as data of the integrated audio object, the audio waveform data and rendering parameters generated by the integration unit, and transmits, as data of the other audio objects, the audio waveform data of each of the other audio objects and the rendering parameters at the predetermined assumed listening position. The information processing device according to (2). (4) The integration unit integrates a plurality of audio objects located at a position that is a predetermined distance or more away from the predetermined assumed listening position. The information processing device according to any one of (1) to (3). (5) The integration unit integrates a plurality of audio objects whose horizontal angles with respect to the predetermined assumed listening position are in a range narrower than a predetermined angle. The information processing device according to any one of (1) to (4). (6) The information processing device according to any one of (1) to (5), wherein the integration unit integrates audio objects that are indistinguishable in sound at the predetermined assumed listening position and that belong to the same group that is set in advance. (7) The integration unit integrates audio objects so that the number of audio objects to be transmitted corresponds to a transmission bit rate. The information processing device according to any one of (1) to (6). (8) The transmitting unit transmits the audio bitstream including flag information indicating whether an audio object included in the audio bitstream is a non-integrated audio object or the integrated audio object. 8. The information processing device according to any one of (1) to (7). (9) The transmitting unit transmits the audio bitstream file together with a playback management file including flag information indicating whether an audio object included in the audio bitstream is an unintegrated audio object or the integrated audio object. 8. The information processing device according to any one of (1) to (7). (10) Integrating audio objects whose sounds cannot be discriminated at a predetermined assumed listening position among a plurality of audio objects for a predetermined assumed listening position among a plurality of assumed listening positions; The data of the integrated audio object obtained by the integration is transmitted together with the data of other audio objects whose sounds can be discriminated at the predetermined assumed listening position. An information processing method comprising the steps of: (11) On the computer, Integrating audio objects whose sounds cannot be discriminated at a predetermined assumed listening position among a plurality of audio objects for a predetermined assumed listening position among a plurality of assumed listening positions; The data of the integrated audio object obtained by the integration is transmitted together with the data of other audio objects whose sounds can be discriminated at the predetermined assumed listening position. A program for executing a process that includes steps. [Explanation of symbols]

[0196] 1 Content generation device, 2 Playback device, 51 Audio encoder, 52 Metadata encoder, 53 Audio generation unit, 54 Video generation unit, 55 Content storage unit, 56 Transmission control unit, 61 Integration unit, 71 Content acquisition unit, 72 Separation unit, 73 Audio playback unit, 74 Video playback unit 73 Audio playback unit

Claims

1. an acquisition unit that acquires data of an integrated audio object obtained by integrating audio objects whose sounds cannot be discriminated at an assumed listening position selected from a plurality of assumed listening positions, and data of other audio objects that have not been integrated; a playback unit for playing back the data of the integrated audio object and the data of the other audio objects; Equipped with The reproduction of the data of the integrated audio object includes a process of reproducing audio waveform data of each integrated audio object based on data of a difference component of the audio waveform data of the integrated audio object. playback device.

2. The acquisition unit acquires, as data of the integrated audio object, audio waveform data and rendering parameters of the integrated audio object that are generated based on audio waveform data and rendering parameters of the multiple audio objects to be integrated. The playback device according to claim 1 .

3. When the flag acquired together with the data of the integrated audio object indicates that the difference component data is present, the reproduction unit performs a process of reproducing the audio waveform data of each integrated audio object based on the difference component data prepared in a generating device for the integrated audio object. The playback device according to claim 1 .

4. The integrated audio object is an audio object obtained by integrating a plurality of audio objects located at a position that is a predetermined distance or more away from the selected assumed listening position.

4. A reproducing apparatus according to claim 1.

5. The integrated audio object is an audio object obtained by integrating a plurality of audio objects whose horizontal angles with respect to the selected assumed listening position are in a range narrower than a predetermined angle.

5. A reproducing apparatus according to claim 1.

6. The integrated audio object is an audio object obtained by integrating audio objects that belong to the same preset group.

4. A reproducing apparatus according to claim 1.

7. an analyzing unit that analyzes whether an audio object included in the stream is a non-integrated audio object or the integrated audio object based on flag information included in the stream; The playback unit plays back data of the integrated audio object when it is analyzed that the audio object included in the stream is the integrated audio object.

7. A reproducing apparatus according to claim 1.

8. an analyzing unit that analyzes whether an audio object included in the stream is an unintegrated audio object or the integrated audio object based on flag information described in a playback management file acquired together with the file of the stream; The playback unit plays back data of the integrated audio object when it is analyzed that the audio object included in the stream is the integrated audio object.

7. A reproducing apparatus according to claim 1.

9. The playback device Acquire data of an integrated audio object obtained by integrating audio objects whose sounds cannot be discriminated at a selected assumed listening position from among the plurality of assumed listening positions, and data of other audio objects that are not integrated; Reproducing the data of the integrated audio object and the data of the other audio objects A regeneration method comprising the steps of: The reproduction of the data of the integrated audio object includes a process of reproducing audio waveform data of each integrated audio object based on data of a difference component of the audio waveform data of the integrated audio object. How to play.

10. On the computer, Acquire data of an integrated audio object obtained by integrating audio objects whose sounds cannot be discriminated at a selected assumed listening position from among the plurality of assumed listening positions, and data of other audio objects that are not integrated; Reproducing the data of the integrated audio object and the data of the other audio objects A program for executing a process, The reproduction of the data of the integrated audio object includes a process of reproducing audio waveform data of each integrated audio object based on data of a difference component of the audio waveform data of the integrated audio object. program.

Images