Information processing device and method, program, and information processing system

Abstract

To provide an information processor and a method, a program and an information processing system that reduce total number of audio objects while minimizing an impact on sound quality.SOLUTION: A pre-rendering processor 11 applicable to an information processing device includes: a priority calculation unit that acquires data of a plurality of audio objects including audio signals and metadata in a space, calculates priority information for each audio object based on the data, and outputs the data including the calculated priority information to a subsequent stage; a pass-through object selection unit that obtains the object data and priority information from the priority calculation unit, and also obtains number information from an external source; and an object generation unit that generates new object metadata and audio signals based on the supplied number information, and the metadata and audio signals of non-pass-through objects supplied by the pass-through object selection section, and outputs them to the subsequent stage.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present technology relates to an information processing device and method, a program, and an information processing system, and in particular to an information processing device and method, a program, and an information processing system that are capable of reducing the total number of objects while minimizing the impact on sound quality. [Background technology]

[0002] Conventionally, the Moving Picture Experts Group (MPEG)-H 3D Audio standard is known (see, for example, Non-Patent Document 1 and Non-Patent Document 2).

[0003] 3D Audio, which is handled by standards such as MPEG-H 3D Audio, can reproduce the direction, distance, and spread of sound in three dimensions, making it possible to reproduce audio with a more realistic feel than conventional stereo playback. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] ISO / IEC 23008-3, MPEG-H 3D Audio [Non-patent document 2] ISO / IEC 23008-3:2015 / AMENDMENT3, MPEG-H 3D Audio Phase 2 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with 3D Audio, if the content contains a large number of objects, the data size of the entire content increases, and the amount of calculation required for decoding and rendering the data for each of the multiple objects also increases. Furthermore, if an upper limit on the number of objects is set for certain operations, it may become impossible to handle content with a number of objects that exceeds that limit.

[0006] Therefore, it is possible to reduce the total number of objects by discarding some of the objects that make up the content. However, in such a case, there is a risk that discarding the objects will degrade the sound quality of the entire content.

[0007] The present technology has been made in view of such circumstances, and makes it possible to reduce the total number of objects while minimizing the impact on sound quality. [Means for solving the problem]

[0008] An information processing device according to a first aspect of the present technology includes a processing unit that acquires data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, and calculates priority information for each of the audio objects based on the data, and the processing unit outputs the data including the calculated priority information to a subsequent stage.

[0009] An information processing method or program according to a first aspect of the present technology includes steps of acquiring data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, calculating priority information for each of the audio objects based on the data, and outputting the data including the calculated priority information to a subsequent stage.

[0010] In a first aspect of the present technology, data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, is acquired, priority information of each of the audio objects is calculated based on the data, and the data including the calculated priority information is output to a subsequent stage.

[0011] An information processing system according to a second aspect of the present technology is an information processing system having an encoding device and a decoding device, wherein the encoding device comprises a processing unit that acquires data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, calculates priority information for each of the audio objects based on the data, and outputs the audio signals of the audio objects and the metadata including the calculated priority information, and an encoding unit that encodes the audio signals of the audio objects and the metadata including the calculated priority information output by the processing unit, and outputs a code string, and the decoding device comprises a decoding unit that decodes the code string to acquire the audio signals of the audio objects and the metadata including the calculated priority information.

[0012] In a second aspect of the present technology, in an information processing system having an encoding device and a decoding device, the encoding device acquires data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, calculates priority information for each of the audio objects based on the data, outputs the audio signals of the audio objects and the metadata including the calculated priority information, encodes the audio signals of the audio objects and the metadata including the calculated priority information output by the processing unit, and outputs a code string. Furthermore, the decoding device decodes the code string to acquire the audio signals of the audio objects and the metadata including the calculated priority information. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 10 is a diagram illustrating how the positions of virtual speakers are determined. [Figure 2] FIG. 1 illustrates an example of the configuration of a pre-rendering processing device. [Figure 3] 10 is a flowchart illustrating an object output process. [Figure 4] FIG. 1 illustrates an example of the configuration of an encoding device. [Figure 5] FIG. 1 illustrates an example of the configuration of an encoding device. [Figure 6] FIG. 10 is a diagram illustrating an example of the configuration of a decoding device. [Figure 7] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0015] First Embodiment About this technology This technology separates multiple objects into pass-through objects and non-pass-through objects, and generates new objects based on the non-pass-through objects, thereby reducing the total number of objects while minimizing the impact on sound quality.

[0016] In the present technology, the object may be any object, such as an audio object or an image object, as long as it has object data.

[0017] The data of the object here refers to, for example, the object signal and metadata of the object.

[0018] Specifically, for example, if the object is an audio object, the audio signal as the object signal and the metadata are the data of the audio object, and if the object is an image object, the image signal as the object signal and the metadata are the data of the image object.

[0019] Hereinafter, the case where the object is an audio object will be described as an example.

[0020] When the object is an audio object, the audio signal of the object and the metadata are handled as the data of the object.

[0021] Here, the metadata includes, for example, position information indicating the position of the object in the three-dimensional space, priority information indicating the priority of the object, gain information of the audio signal of the object, spread information indicating the spread of the sound image of the object, and the like.

[0022] Also, the position information of the object consists of, for example, a radius indicating the distance from the reference position to the object, a horizontal angle indicating the horizontal position of the object, and a vertical angle indicating the vertical position of the object.

[0023] This technology can be applied to, for example, a prerendering processing device that takes as input a plurality of objects constituting content, more specifically, the data of the objects, and outputs an appropriate number of objects, more specifically, the data of the objects, in response to the input.

[0024] Hereinafter, the number of objects at the time of input is denoted as nobj_in, and the number of objects at the time of output is denoted as nobj_out. In particular, here nobj_out < nobj_in. That is, the number of output objects is made less than the number of input objects.

[0025] In this technology, some of the input nobj_in objects are output as data without any changes, that is, as passed-through objects. Hereinafter, such passed-through objects will be referred to as pass-through objects.

[0026] Furthermore, among the input nobj_in objects, objects that have not been set as pass-through objects are set as non-pass-through objects. In this technology, data of the non-pass-through objects is used to generate data of a new object.

[0027] When nobj_in objects are input in this way, the objects are separated into pass-through objects and non-pass-through objects.

[0028] Then, based on the objects determined to be non-pass-through objects, new objects are generated in a number smaller than the total number of the non-pass-through objects, and data of the generated new objects and data of the pass-through objects are output.

[0029] In this way, in this technology, nobj_out objects, which are fewer than the input nobj_in objects, are output, thereby realizing a reduction in the total number of objects.

[0030] In the following, the number of objects that are set as pass-through objects is assumed to be nobj_dynamic. For example, the number of pass-through objects, nobj_dynamic, can be set by the user or the like within a range that satisfies the condition shown in the following formula (1).

[0031]

number

[0032] From the condition shown in equation (1), the number of pass-through objects nobj_dynamic is set to be equal to or greater than 0 and less than nobj_out.

[0033] For example, the number of pass-through objects, nobj_dynamic, can be a predetermined number or a number specified by a user's input operation, etc. However, the number of pass-through objects, nobj_dynamic, may be dynamically determined so as to be equal to or less than a predetermined maximum number based on the data volume (data size) of the entire content, the amount of calculation required for the decoding process, etc. In this case, the predetermined maximum number is set to a number less than nobj_out.

[0034] The data amount of the entire content is the total data amount (data size) of the metadata and audio signals of the pass-through object and the metadata and audio signals of the newly generated object. The computational amount of the decoding process taken into consideration when determining the number nobj_dynamic may be the computational amount of only the decoding process of the encoded data (metadata and audio signals) of the object, or may be the total computational amount of the decoding process and the rendering process.

[0035] In addition to the number of pass-through objects nobj_dynamic, the number of objects nobj_out that are finally output may also be determined based on the data volume of the entire content or the amount of calculation required for the decoding process, or the number nobj_out may be specified by the user, etc. Furthermore, the number nobj_out may be determined in advance.

[0036] Here, a specific example of a method for selecting a pass-through object will be described.

[0037] First, in the following, an index indicating a time frame of an audio signal is referred to as ifrm, and an index indicating an object is referred to as iobj. Note that, in the following, a time frame whose index is ifrm will also be referred to as time frame ifrm, and an object whose index is iobj will also be referred to as object iobj.

[0038] Furthermore, priority information is included in the metadata for each object, and the priority information included in the metadata for the time frame ifrm of object iobj is denoted as priority_raw[ifrm][iobj]. In other words, the priority information priority_raw[ifrm][iobj] is included in the metadata previously assigned to the object.

[0039] In such a case, for example, in the present technology, the value of priority information priority[ifrm][iobj] shown in the following equation (2) is calculated for each object for each time frame.

[0040]

number

[0041] In the formula (2), priority_gen[ifrm][iobj] is priority information of the time frame ifrm of the object iobj, which is calculated based on information other than priority_raw[ifrm][iobj].

[0042] For example, the priority information priority_gen[ifrm][iobj] can be calculated using gain information, position information, spread information included in the metadata, as well as audio signals of objects, either singly or in any combination. Furthermore, the priority information priority_gen[ifrm][iobj] of the current time frame may be calculated using not only the gain information, position information, spread information, and audio signals of the current time frame, but also the gain information, position information, spread information, and audio signals of a time frame immediately preceding the current time frame.

[0043] A specific method for calculating the priority information priority_gen[ifrm][iobj] may be, for example, the method described in International Publication No. 2018 / 198789.

[0044] That is, for example, the reciprocal of the radius constituting the position information included in the metadata can be set as the priority information priority_gen[ifrm][iobj] so that the closer an object is to the user, the higher the priority.Furthermore, for example, the closer an object is to the user, the higher the priority can be set as the priority information priority_gen[ifrm][iobj] so that the reciprocal of the absolute value of the horizontal angle constituting the position information included in the metadata can be set as the priority information priority_gen[ifrm][iobj].

[0045] Furthermore, the object's movement speed may be used as priority information priority_gen[ifrm][iobj] based on the position information contained in the metadata of different time frames, or the gain information contained in the metadata itself may be used as priority information priority_gen[ifrm][iobj].

[0046] Alternatively, the priority information priority_gen[ifrm][iobj] may be, for example, the squared value of the spread information included in the metadata, or the priority information priority_gen[ifrm][iobj] may be calculated based on the attribute information of the object.

[0047] Furthermore, in equation (2), weight is a parameter that determines the ratio between the priority information priority_raw[ifrm][iobj] and the priority information priority_gen[ifrm][iobj] in calculating the priority information priority[ifrm][iobj], and is set to, for example, 0.5.

[0048] In addition, in the MPEG-H 3D Audio standard, priority information priority_raw[ifrm][iobj] may not be assigned to an object, so in such cases the value of the priority information priority_raw[ifrm][iobj] is set to 0 and the calculation of equation (2) is performed.

[0049] When the priority information priority[ifrm][iobj] for each object is calculated using equation (2), the priority information priority[ifrm][iobj] for each object is sorted in descending order for each time frame ifrm. The top nobj_dynamic number of objects with the highest priority information priority[ifrm][iobj] values are selected as pass-through objects for the time frame ifrm, and the remaining objects are non-pass-through objects.

[0050] In other words, by selecting nobj_dynamic objects in descending order of priority information priority[ifrm][iobj], nobj_in objects are separated into nobj_dynamic pass-through objects and (nobj_in-nobj_dynamic) non-pass-through objects.

[0051] After the separation, for the nobj_dynamic number of pass-through objects, the metadata and audio signals of those pass-through objects are output to the subsequent stage as is.

[0052] On the other hand, for the (nobj_in-nobj_dynamic) non-pass-through objects, a rendering process, i.e., a pre-rendering process, is performed on those non-pass-through objects, thereby generating metadata and audio signals for new (nobj_out-nobj_dynamic) objects.

[0053] Specifically, for example, a rendering process using VBAP (Vector Based Amplitude Panning) is performed on each non-pass-through object, and the non-pass-through object is rendered into (nobj_out-nobj_dynamic) virtual speakers. Here, the virtual speakers correspond to new objects, and the placement positions of these virtual speakers in three-dimensional space are set to be different from each other.

[0054] For example, let spk be the index indicating a virtual speaker, and let the virtual speaker indicated by the index spk be referred to as virtual speaker spk. Also, let sig[ifrm][iobj] be the audio signal in the time frame ifrm of a non-pass-through object whose index is iobj.

[0055] In this case, for each non-pass-through object iobj, VBAP is performed based on the position information included in the metadata and the position of the virtual speaker in the three-dimensional space. As a result, for each non-pass-through object iobj, the gains gain[ifrm][iobj][spk] of each of the (nobj_out-nobj_dynamic) virtual speakers spk are obtained.

[0056] Then, for each virtual speaker spk, the sum of the audio signals sig[ifrm][iobj] multiplied by the gain gain[ifrm][iobj][spk] of the virtual speaker spk for each non-pass-through object iobj is calculated, and the resulting audio signal is used as the audio signal of the new object corresponding to that virtual speaker spk.

[0057] For example, the position of a virtual speaker corresponding to a new object is determined by the k-means method. That is, the position information included in the metadata of the non-pass-through object for each time frame is divided into (nobj_out-nobj_dynamic) clusters by the k-means method, and the position of the center of gravity of each cluster is determined as the position of the virtual speaker.

[0058] Therefore, when nobj_in=24, nobj_dynamic=5, and nobj_out=10, the positions of the virtual speakers are calculated as shown in Fig. 1. In this case, the positions of the virtual speakers may change depending on the time frame.

[0059] In Figure 1, circles without hatching (diagonal lines) represent non-pass-through objects, and these non-pass-through objects are located at positions in three-dimensional space indicated by the position information contained in the metadata.

[0060] In this example, the above-mentioned classification is performed for each time frame, nobj_dynamic (=5) pass-through objects are selected, and the remaining (nobj_in-nobj_dynamic (=24-5=19)) objects are set as non-pass-through objects.

[0061] Here, since the number of virtual speakers (nobj_out-nobj_dynamic) is 10-5=5, the position information of the 19 non-pass-through objects is divided into five clusters, and the center of gravity position of each cluster is set to the position of virtual speakers SP11-1 to SP11-5.

[0062] 1, virtual speakers SP11-1 to SP11-5 are arranged at the center of gravity of the clusters corresponding to those virtual speakers. Note that, hereinafter, when there is no need to particularly distinguish between virtual speakers SP11-1 to SP11-5, they will also be simply referred to as virtual speaker SP11.

[0063] In the rendering process, the 19 non-pass-through objects are rendered onto the five virtual speakers SP11 thus obtained.

[0064] Note that the rendering process determines the audio signal of the new object corresponding to the virtual speaker SP11, and the position information included in the metadata of the new object is information indicating the position of the virtual speaker SP11 corresponding to the new object.

[0065] Furthermore, information other than the position information included in the metadata of the new object, such as priority information, gain information, and spread information, is set to the average value or maximum value of the information in the metadata of the non-pass-through objects included in the cluster corresponding to the new object. That is, for example, the average value or maximum value of the gain information of the non-pass-through objects belonging to the cluster is set to the gain information included in the metadata of the new object corresponding to the cluster.

[0066] When the audio signals and metadata of (nobj_out-nobj_dynamic=5) new objects are generated in this manner, the audio signals and metadata of these new objects are output to the subsequent stage.

[0067] Therefore, in this example, the audio signals and metadata of (nobj_dynamic=5) pass-through objects and the audio signals and metadata of (nobj_out-nobj_dynamic=5) new objects will be output to the subsequent stage.

[0068] In other words, audio signals and metadata of a total of (nobj_out=10) objects will be output.

[0069] In this way, nobj_out objects, which are fewer than the nobj_in objects input, are output, thereby reducing the total number of objects.

[0070] This reduces the data size of the entire content consisting of multiple objects, and also reduces the amount of calculation required for decoding and rendering the objects in later stages. Furthermore, even if the number of input objects, nobj_in, exceeds the number of objects specified by the operation, the number of output objects can be set to the number specified by the operation, so that the content consisting of the data of the output objects can be handled in the operation.

[0071] Furthermore, in this technology, objects with high priority information priority[ifrm][iobj] are treated as pass-through objects and the audio signal and metadata are output as is, so there is no degradation in the sound quality of the content audio for pass-through objects.

[0072] Furthermore, for non-pass-through objects, new objects are generated based on those non-pass-through objects, so the impact on the sound quality of the content can be minimized. In particular, if a new object is generated using non-pass-through objects, the sound of the content will contain sound components from all objects.

[0073] Therefore, compared to, for example, a case where only a manageable number of objects are left and the other objects are discarded, it is possible to keep the impact on the sound quality of the audio of the content low.

[0074] As described above, according to the present technology, it is possible to reduce the total number of objects while minimizing the impact on sound quality.

[0075] Although the above description has been given of an example in which the positions of the virtual speakers are determined by the k-means method, the positions of the virtual speakers may be determined in any manner.

[0076] For example, the non-pass-through objects may be grouped (clustered) by a method other than the k-means method according to the concentration of the non-pass-through objects in the three-dimensional space, and the position of the center of gravity of each group or the average position of the positions of the non-pass-through objects belonging to the group may be set as the position of the virtual speaker. Note that the concentration of objects in the three-dimensional space indicates how concentrated (densely packed) the objects are arranged in the three-dimensional space.

[0077] Furthermore, the number of groups at the time of grouping may be determined according to the concentration of non-pass-through objects so as to be a predetermined number less than (nobj_in-nobj_dynamic).

[0078] Alternatively, even when the k-means method is used, the number of newly generated objects may be determined so as to be equal to or less than a predetermined maximum number depending on the concentration of the positions of non-pass-through objects, the user's operation to specify the number, the data volume (data size) of the entire content, and the amount of calculation required for the decoding process. In such cases, the number of newly generated objects only needs to be less than (nobj_in-nobj_dynamic), which satisfies the condition of the above-mentioned formula (1).

[0079] The positions of the virtual speakers may also be set to predetermined fixed positions. In this case, for example, if the position of each virtual speaker is set to the position of each speaker in a 22-channel speaker arrangement, it will be easier to handle new objects in a later stage. Alternatively, the positions of some of the multiple virtual speakers may be set to predetermined fixed positions, and the positions of the remaining virtual speakers may be determined using a k-means method or the like.

[0080] Furthermore, although an example will be described here in which all objects that are not designated as pass-through objects are designated as non-pass-through objects, there may be objects that are discarded without being designated as either pass-through objects or non-pass-through objects. In such a case, for example, a predetermined number of lower-ranking objects with small values of priority information priority[ifrm][iobj] may be discarded, or objects whose value of priority information priority[ifrm][iobj] is equal to or less than a predetermined threshold may be discarded.

[0081] For example, if the content is made up of multiple objects, such as the audio of a movie, some of the objects may be so unimportant that discarding them will have little effect on the audio quality of the resulting content. Therefore, in such cases, even if some of the objects that were not designated as pass-through objects are designated as non-pass-through objects, there will be almost no effect on the audio quality.

[0082] On the other hand, when the content consisting of multiple objects is, for example, music, in most cases it does not contain objects of low importance, so it is important to make all objects that have not been designated as pass-through objects non-pass-through objects in order to minimize the impact on sound quality.

[0083] Although the above description has been given of an example in which pass-through objects are selected based on priority information, pass-through objects may also be selected based on the degree of concentration (density) of objects in a three-dimensional space.

[0084] In such cases, objects are grouped based on, for example, location information included in the metadata of each object, and the objects are then classified based on the grouping results.

[0085] Specifically, for example, an object whose distance from any other object is greater than a predetermined value can be considered a pass-through object, and an object whose distance from any other object is less than the predetermined value can be considered a non-pass-through object.

[0086] Furthermore, clustering (grouping) may be performed using a k-means method or the like based on the location information contained in the metadata of each object, and if only one object belongs to a cluster, the object belonging to that cluster may be considered a pass-through object.

[0087] In this case, for a cluster to which multiple objects belong, all objects belonging to the cluster may be designated as non-pass-through objects, or the object with the highest priority indicated by priority information among the objects belonging to the cluster may be designated as a pass-through object, and the remaining objects may be designated as non-pass-through objects.

[0088] Even when pass-through objects are selected based on the degree of concentration, etc., the number of pass-through objects nobj_dynamic may be dynamically determined depending on the results of grouping or clustering, the amount of data (data size) of the entire content, the amount of calculation required for the decoding process, etc.

[0089] Furthermore, instead of generating a new object by rendering using VBAP or the like, the audio signal for the new object may be an average value or a linear combination value of the audio signals of non-pass-through objects. The technique of generating a new object using an average value or the like is particularly useful when there is only one newly generated object.

[0090] <Configuration Example of Pre-rendering Processing Device> Next, a pre-rendering processing device to which the above-described present technology is applied will be described. Such a pre-rendering processing device may be configured, for example, as shown in FIG.

[0091] The pre-rendering processing device 11 shown in Figure 2 is an information processing device that receives data of multiple objects as input and outputs data of fewer objects than the input, and has a priority calculation unit 21, a pass-through object selection unit 22, and an object generation unit 23.

[0092] In this pre-rendering processing device 11, data of nobj_in objects, that is, object metadata and audio signals, are supplied to a priority calculation unit 21.

[0093] Further, the pass-through object selection unit 22 and the object generation unit 23 are supplied with number information indicating the number of input objects nobj_in, the number of output objects nobj_out, and the number of pass-through objects nobj_dynamic.

[0094] The priority calculation unit 21 calculates priority information priority[ifrm][iobj] of each object based on the metadata and audio signals of the supplied objects, and supplies the priority information priority[ifrm][iobj], metadata, and audio signals of each object to the pass-through object selection unit 22.

[0095] The pass-through object selection unit 22 is supplied with object metadata, audio signals, and priority information priority[ifrm][iobj] from the priority calculation unit 21, and also with number information from an external source. In other words, the pass-through object selection unit 22 acquires object data and priority information priority[ifrm][iobj] from the priority calculation unit 21, and also acquires number information from an external source.

[0096] The pass-through object selection unit 22 selects a pass-through object based on the supplied number information and the priority information priority[ifrm][iobj] supplied from the priority calculation unit 21. The pass-through object selection unit 22 outputs the metadata and audio signals of the pass-through objects supplied from the priority calculation unit 21 to a subsequent stage as is, and supplies the metadata and audio signals of the non-pass-through objects supplied from the priority calculation unit 21 to the object generation unit 23.

[0097] The object generation unit 23 generates metadata and audio signals for new objects based on the supplied number information and the metadata and audio signals for non-pass-through objects supplied from the pass-through object selection unit 22, and outputs them to the subsequent stage.

[0098] <Explanation of Object Output Processing> Next, a description will be given of the operation of the pre-rendering processing device 11. That is, the object output processing by the pre-rendering processing device 11 will be described below with reference to the flowchart in FIG.

[0099] In step S11, the priority calculation unit 21 calculates priority information priority[ifrm][iobj] of each object based on the metadata and audio signal of each object in the given time frame that has been supplied.

[0100] For example, the priority calculation unit 21 calculates priority information priority_gen[ifrm][iobj] for each object based on the metadata and audio signal, and calculates the priority information priority[ifrm][iobj] using equation (2) based on the priority information priority_raw[ifrm][iobj] included in the metadata and the calculated priority information priority_gen[ifrm][iobj].

[0101] The priority calculation unit 21 supplies the priority information priority[ifrm][iobj], metadata, and audio signals of each object to the pass-through object selection unit 22 .

[0102] In step S12, the pass-through object selection unit 22 selects nobj_dynamic pass-through objects from among the nobj_in objects based on the supplied number information and the priority information priority[ifrm][iobj] supplied from the priority calculation unit 21. In other words, the objects are classified.

[0103] Specifically, the pass-through object selection unit 22 sorts the priority information priority[ifrm][iobj] of each object and selects the top nobj_dynamic number of objects with the largest values of priority information priority[ifrm][iobj] as pass-through objects. In this case, all of the input nobj_in objects that have not been selected as pass-through objects are selected as non-pass-through objects, but only some of the objects that are not pass-through objects may be selected as non-pass-through objects.

[0104] In step S13, the pass-through object selection unit 22 outputs the metadata and audio signals of the pass-through object selected in the processing of step S12 from the metadata and audio signals of each object supplied from the priority calculation unit 21 to the subsequent stage.

[0105] The pass-through object selection unit 22 also supplies the object generation unit 23 with the metadata and audio signals of the (nobj_in-nobj_dynamic) non-pass-through objects obtained by classifying the objects.

[0106] Although an example in which objects are separated based on priority information will be described here, pass-through objects may also be selected based on the degree of concentration of object positions, as described above.

[0107] In step S14, the object generation unit 23 determines the positions of (nobj_out-nobj_dynamic) virtual speakers based on the metadata and audio signals of the non-pass-through objects supplied from the pass-through object selection unit 22 and the supplied number information.

[0108] For example, the object generation unit 23 clusters the position information of non-pass-through objects using the k-means method, and sets the center of gravity positions of each of the resulting (nobj_out-nobj_dynamic) clusters as the positions of the virtual speakers corresponding to those clusters.

[0109] The method for determining the positions of the virtual speakers is not limited to the k-means method, and other methods may be used, or predetermined fixed positions may be set as the positions of the virtual speakers.

[0110] In step S15, the object generation unit 23 performs rendering processing based on the metadata and audio signals of the non-pass-through objects supplied from the pass-through object selection unit 22 and the positions of the virtual speakers obtained in step S14.

[0111] For example, the object generation unit 23 performs VBAP as a rendering process to obtain the gain gain[ifrm][iobj][spk] of each virtual speaker. The object generation unit 23 also obtains the sum of the audio signals sig[ifrm][iobj] of the non-pass-through objects multiplied by the gain gain[ifrm][iobj][spk] for each virtual speaker, and sets the resulting audio signal as the audio signal of a new object corresponding to the virtual speaker.

[0112] Furthermore, the object generation unit 23 generates metadata for a new object based on the clustering results obtained when determining the positions of the virtual speakers and the metadata of the non-pass-through objects.

[0113] As a result, metadata and audio signals for (nobj_out-nobj_dynamic) new objects can be obtained. Note that the method for generating audio signals for new objects may be a rendering process other than VBAP.

[0114] In step S16, the object generating unit 23 outputs the metadata and audio signals of the (nobj_out-nobj_dynamic) new objects obtained in the process of step S15 to the subsequent stage.

[0115] As a result, for one time frame, metadata and audio signals of nobj_dynamic pass-through objects and metadata and audio signals of (nobj_out-nobj_dynamic) new objects are output.

[0116] That is, the metadata and audio signals of a total of nobj_out objects are output as the metadata and audio signals of the pre-rendered objects.

[0117] In step S17, the pre-rendering processing device 11 determines whether or not the processing has been performed for all time frames.

[0118] If it is determined in step S17 that the processing has not yet been performed for all the time frames, the processing returns to step S11, and the above-described processing is repeated, that is, the processing is performed for the next time frame.

[0119] On the other hand, if it is determined in step S17 that the processing has been performed for all time frames, each unit of the pre-rendering processing device 11 stops the processing it is performing, and the object output processing ends.

[0120] In this way, the pre-rendering processing device 11 classifies objects based on priority information, outputs the metadata and audio signals as they are for pass-through objects with high priority, and performs rendering processing for non-pass-through objects to generate and output the metadata and audio signals of new objects.

[0121] Therefore, for objects with high priority information that have a large impact on the sound quality of the content's audio, the metadata and audio signal are output as is, and for other objects, new objects are generated through rendering processing, thereby reducing the total number of objects while minimizing the impact on sound quality.

[0122] Although an example in which objects are classified for each time frame has been described above, the same object may always be set as a pass-through object regardless of the time frame.

[0123] In such a case, for example, the priority calculation unit 21 obtains the priority information priority[ifrm][iobj] for all time frames for the object, and sets the sum of the priority information priority[ifrm][iobj] obtained for all time frames as the priority information priority[iobj] of the object.The priority calculation unit 21 then sorts the priority information priority[iobj] of each object and selects the top nobj_dynamic number of objects with the largest priority information priority[iobj] values as pass-through objects.

[0124] Alternatively, objects may be classified for each interval consisting of multiple consecutive time frames. In such a case, the priority information of each object for each interval can be calculated in the same way as the priority information priority[iobj].

[0125] <Example 1 of application of this technology to an encoding device> <Configuration example of encoding device> The present technology described above can be applied to an encoding device having a 3D Audio encoding unit that encodes 3D Audio. Such an encoding device may be configured, for example, as shown in FIG. 4.

[0126] The encoding device 51 shown in FIG. 4 includes a pre-rendering processing unit 61 and a 3D audio encoding unit 62.

[0127] 2, and has the same configuration as the pre-rendering processing device 11. That is, the pre-rendering processing device 61 has the priority calculation unit 21, pass-through object selection unit 22, and object generation unit 23 described above.

[0128] Metadata and audio signals of multiple objects are supplied to the pre-rendering processing unit 61. The pre-rendering processing unit 61 performs pre-rendering processing to reduce the total number of objects, and supplies the metadata and audio signals of each reduced object to the 3D Audio encoding unit 62.

[0129] The 3D Audio encoding unit 62 encodes the object metadata and audio signal supplied from the pre-rendering processing unit 61, and outputs the resulting 3D Audio code string.

[0130] For example, it is assumed that the pre-rendering processing unit 61 is supplied with metadata and audio signals of nobj_in objects.

[0131] In this case, the pre-rendering processing unit 61 performs processing similar to the object output processing described with reference to FIG. 3, and supplies the metadata and audio signals of the nobj_dynamic number of pass-through objects and the metadata and audio signals of the (nobj_out-nobj_dynamic) number of new objects to the 3D Audio encoding unit 62.

[0132] Therefore, in this example, the 3D Audio encoding unit 62 encodes and outputs the metadata and audio signals of a total of nobj_out objects.

[0133] In this way, the encoding device 51 reduces the total number of objects and encodes each of the reduced objects. This reduces the size (code amount) of the output 3D Audio codestring, as well as the amount of calculation and memory required for the encoding process. Furthermore, on the decoding side of the 3D Audio codestring, it also reduces the amount of calculation and memory required in the 3D Audio decoding unit that decodes the 3D Audio codestring and in the subsequent rendering processing unit.

[0134] Note that the example described here is one in which the pre-rendering processing unit 61 is arranged inside the encoding device 51. However, the present invention is not limited to this, and the pre-rendering processing unit 61 may be arranged outside the encoding device 51, i.e., before the encoding device 51, or may be arranged at the very front end inside the 3D Audio encoding unit 62.

[0135] <Example 2 of application of this technology to an encoding device> <Configuration example of encoding device> Furthermore, when the present technology is applied to an encoding device, a pre-rendering processing flag indicating whether an object is a pass-through object or a newly generated object may also be included in the 3D Audio codestream.

[0136] In such a case, the encoding device may be configured as shown in Fig. 5. In Fig. 5, the same reference numerals are used to designate parts that correspond to those in Fig. 4, and the description thereof will be omitted where appropriate.

[0137] The encoding device 91 shown in FIG. 5 includes a pre-rendering processing unit 101 and a 3D audio encoding unit 62.

[0138] 2, and has the same configuration as the pre-rendering processing device 11. That is, the pre-rendering processing device 101 has the priority calculation unit 21, pass-through object selection unit 22, and object generation unit 23 described above.

[0139] However, in the pre-rendering processing unit 101, the pass-through object selection unit 22 and the object generation unit 23 generate a pre-rendering processing flag for each object, and output metadata, an audio signal, and a pre-rendering processing flag for each object.

[0140] The pre-rendering processing flag is flag information indicating whether the object is a pass-through object or a newly generated object, that is, whether the object is a pre-rendered object.

[0141] For example, if an object is a pass-through object, the value of the pre-rendering flag for that object is set to 0. Conversely, if the object is a newly generated object, the value of the pre-rendering flag for that object is set to 1.

[0142] Therefore, for example, the pre-rendering processing unit 101 performs processing similar to the object output processing described with reference to FIG. 3 to reduce the total number of objects, and generates a pre-rendering processing flag for each object after the total number has been reduced.

[0143] The pre-rendering processing unit 101 then supplies the metadata, audio signals, and pre-rendering processing flags with a value of 0 for the nobj_dynamic number of pass-through objects to the 3D Audio encoding unit 62.

[0144] In response to this, the pre-rendering processing unit 101 supplies the metadata, audio signals, and pre-rendering processing flags with a value of 1 for the (nobj_out-nobj_dynamic) new objects to the 3D Audio encoding unit 62.

[0145] The 3D Audio encoding unit 62 encodes the metadata, audio signals, and pre-rendering processing flags of a total of nobj_out objects supplied from the pre-rendering processing unit 101, and outputs the resulting 3D Audio code string.

[0146] <Configuration example of a decoding device> A decoding device that receives as input the 3D Audio code string that includes the pre-rendering processing flag and is output from the encoding device 91 and performs decoding is configured as shown in FIG. 6, for example.

[0147] The decoding device 131 shown in FIG. 6 includes a 3D audio decoding unit 141 and a rendering processing unit 142.

[0148] The 3D Audio decoding unit 141 acquires, by reception or the like, the 3D Audio code string output from the encoding device 91, decodes the acquired 3D Audio code string, and supplies the resulting object metadata, audio signal, and pre-rendering processing flag to the rendering processing unit 142.

[0149] The rendering processing unit 142 performs rendering processing based on the metadata, audio signal, and pre-rendering processing flag supplied from the 3D Audio decoding unit 141, and generates and outputs speaker drive signals for each speaker used to play back the content. These speaker drive signals are signals for playing back the sound of each object that makes up the content through the speakers.

[0150] In the decoding device 131 configured as above, the use of a pre-rendering processing flag can reduce the amount of calculation and memory required for processing in the 3D Audio decoding unit 141 and the rendering processing unit 142. In particular, in this example, the amount of calculation and memory required during decoding can be further reduced compared to the case of the encoding device 51 shown in FIG.

[0151] Here, a specific example of the use of the pre-rendering process flag in the 3D Audio decoding unit 141 and the rendering process unit 142 will be described.

[0152] First, an example of how the pre-rendering process flag is used in the 3D Audio decoding unit 141 will be described.

[0153] The 3D Audio codestream contains object metadata, audio signals, and pre-rendering flags. As mentioned above, the metadata contains priority information, but in some cases the metadata does not contain priority information. The priority information here refers to the priority information priority_raw[ifrm][iobj] mentioned above.

[0154] The value of the pre-rendering process flag is set based on the priority information priority[ifrm][iobj] calculated by the pre-rendering process unit 101, which is located before the 3D Audio encoding unit 62. Therefore, for example, a pass-through object whose pre-rendering process flag has a value of 0 can be said to be a high-priority object, and a newly generated object whose pre-rendering process flag has a value of 1 can be said to be a low-priority object.

[0155] Therefore, in the 3D Audio decoding unit 141, if the metadata does not include priority information, the pre-rendering processing flag can be used instead of the priority information.

[0156] Specifically, for example, it is assumed that the 3D Audio decoding unit 141 decodes only objects with high priority.

[0157] At this time, for example, if the value of the pre-rendering process flag for an object is 1, the 3D Audio decoding unit 141 determines that the value of the priority information for that object is 0, and does not decode the audio signals, etc., included in the 3D Audio codestream for that object.

[0158] On the other hand, if the value of the pre-rendering process flag for an object is 0, the 3D Audio decoding unit 141 determines that the value of the priority information for that object is 1, and decodes the metadata and audio signals included in the 3D Audio codestream for that object.

[0159] In this way, the amount of decoding calculations and memory required can be reduced by the amount of objects for which the decoding process is omitted. Note that the pre-rendering processing unit 101 of the encoding device 91 may generate metadata priority information based on the pre-rendering process flag, i.e., the selection result of the pass-through object.

[0160] Next, an example of how the pre-rendering process flag is used in the rendering process unit 142 will be described.

[0161] The rendering processing unit 142 may perform spreading processing based on spreading information included in the metadata.

[0162] Here, the spread processing is a process for widening the sound image of the sound of an object based on the value of spread information included in the metadata for each object, and is used to enhance the sense of realism.

[0163] On the other hand, an object whose pre-rendering processing flag has a value of 1 is an object newly generated by the pre-rendering processing unit 101 of the encoding device 91, i.e., an object that is a mixture of multiple objects that have been determined to be non-pass-through objects. The value of the spread information of such a newly generated object is a single value calculated by averaging the spread information of the multiple non-pass-through objects.

[0164] Therefore, when spreading is performed on an object whose pre-rendering flag value is 1, the spreading process is performed on what originally were multiple objects based on a single piece of spread information that may not be appropriate, which may result in a reduced sense of realism.

[0165] Therefore, the rendering processing unit 142 can perform spread processing based on spread information for objects whose pre-rendering processing flag has a value of 0, and not perform spread processing for objects whose pre-rendering processing flag has a value of 1. This prevents a decrease in the sense of realism, and also reduces the amount of calculation and memory required by not performing unnecessary spread processing.

[0166] Additionally, a pre-rendering processing device to which the present technology is applied may be provided in a device that plays or edits content consisting of multiple objects, a decoding device, etc. For example, in an application program that edits tracks corresponding to objects, if the number of tracks is too large, editing becomes cumbersome, so it is effective to apply the present technology, which can reduce the number of tracks, i.e., the number of objects, during editing.

[0167] <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0168] FIG. 7 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0169] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0170] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0171] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0172] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0173] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0174] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0175] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0176] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0177] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0178] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0179] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0180] Furthermore, the present technology can also be configured as follows.

[0181] (1) a pass-through object selection unit that acquires data of L objects and selects M pass-through objects from the L objects to output the data as is; an object generation unit that generates the data of N new objects, which is less than (LM), based on the data of a plurality of non-pass-through objects that are not the pass-through objects among the L objects; An information processing device comprising: (2) The object generation unit generates the data of the new object based on the data of the (LM) non-pass-through objects. The information processing device described in (1). (3) The object generation unit generates the data of the N new objects to be placed at mutually different positions by rendering processing based on the data of the plurality of non-pass-through objects. An information processing device according to (1) or (2). (4) The object generation unit determines positions of the N new objects based on position information included in the data of the plurality of non-pass-through objects. (3) An information processing device according to the present invention. (5) The object generation unit determines the positions of the N new objects using a k-means method based on the position information. (4) An information processing device according to the present invention. (6) The positions of the N new objects are set to predetermined positions. (3) An information processing device according to the present invention. (7) The data is an object signal and metadata of the object. An information processing device according to any one of (3) to (6). (8) The object is an audio object (7) An information processing device according to (7). (9) The object generation unit performs VBAP as the rendering process. (8) An information processing device according to (8). (10) The pass-through object selection unit selects the M pass-through objects based on priority information of the L objects. An information processing device according to any one of (1) to (9). (11) The pass-through object selection unit selects the M pass-through objects based on the concentration of the L objects in space. An information processing device according to any one of (1) to (9). (12) The number M of pass-through objects is a specified number. An information processing device according to any one of (1) to (11). (13) The pass-through object selection unit determines the number M of the pass-through objects based on a total data size of the data of the pass-through objects and the data of the new object. An information processing device according to any one of (1) to (11). (14) The pass-through object selection unit determines the number M of pass-through objects based on the amount of calculation required for processing when the data of the pass-through objects and the data of the new objects are decrypted. An information processing device according to any one of (1) to (11). (15) The information processing device Get the data of L objects, Select M pass-through objects from the L objects that output the data as is, generating the data of N new objects, which is less than (LM), based on the data of a plurality of non-pass-through objects that are not the pass-through objects among the L objects; Information processing methods. (16) Get the data of L objects, Select M pass-through objects from the L objects that output the data as is, generating the data of N new objects, which is less than (LM), based on the data of a plurality of non-pass-through objects that are not the pass-through objects among the L objects; A program that causes a computer to execute a process that includes steps. [Explanation of symbols]

[0182] 11 Pre-rendering processing device, 21 Priority calculation unit, 22 Pass-through object selection unit, 23 Object generation unit

Claims

1. a processing unit that acquires data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, and calculates priority information of each of the audio objects based on the data; The processing unit a priority calculation unit that calculates the priority information of the audio object based on the acquired data, and outputs the data including the audio signal and the metadata including the calculated priority information to a subsequent stage; a pass-through object selection unit that selects a pass-through object from among the plurality of audio objects based on the audio signal output from the priority calculation unit and the metadata including the calculated priority information; and The pass-through object selection unit outputs the audio signal of the pass-through object and the metadata including the calculated priority information to a subsequent stage as they are. Information processing device.

2. The metadata includes at least one of position information, priority information, gain information, and spread information of the audio object. The information processing device according to claim 1 .

3. The priority calculation unit calculates the priority information based on the audio signal and the metadata. The information processing device according to claim 1 .

4. the pass-through object selection unit outputs the audio signal of a non-pass-through object that is not the pass-through object and the metadata including the calculated priority information; The audio signal processing device further includes an object generating unit that generates the audio signal of a new audio object and the metadata including the priority information based on the audio signal of the non-pass-through object output by the pass-through object selecting unit and the metadata including the calculated priority information, and outputs the generated audio signal and the metadata including the priority information to a subsequent stage. The information processing device according to claim 1 .

5. an encoding unit that encodes the audio signal and the metadata of the audio object output by the processing unit and outputs a code string; The information processing device according to claim 1 .

6. The audio signal processing device further includes a decoding unit that decodes the code string to obtain the audio signal of the audio object obtained by the decoding and the metadata including the calculated priority information. The information processing device according to claim 1 .

7. a decoding unit that decodes a code string to obtain the data of the audio object obtained by the decoding, The processing unit calculates the priority information based on the data of the audio object acquired by the decoding unit. The information processing device according to claim 1 .

8. The information processing device acquiring data for a plurality of audio objects in a space, the data including audio signals and metadata for the audio objects; calculating priority information for each of the audio objects based on the data, and outputting the data including the audio signal and the metadata including the calculated priority information; selecting a pass-through object from among the plurality of audio objects based on the output audio signal and the metadata including the calculated priority information; The audio signal of the pass-through object and the metadata including the calculated priority information are output as they are to a subsequent stage. Information processing methods.

9. acquiring data for a plurality of audio objects in a space, the data including audio signals and metadata for the audio objects; calculating priority information for each of the audio objects based on the data, and outputting the data including the audio signal and the metadata including the calculated priority information; selecting a pass-through object from among the plurality of audio objects based on the output audio signal and the metadata including the calculated priority information; The audio signal of the pass-through object and the metadata including the calculated priority information are output as they are to a subsequent stage. A program that causes a computer to execute a process that includes steps.

10. An information processing system having an encoding device and a decoding device, The encoding device a processing unit that acquires data of a plurality of audio objects in a space, the data including audio signals and metadata of the audio objects, calculates priority information for each of the audio objects based on the data, and outputs the audio signals of the audio objects and the metadata including the calculated priority information; an encoding unit that encodes the audio signal of the audio object and the metadata including the calculated priority information output by the processing unit, and outputs a code string; Preparation, The processing unit a priority calculation unit that calculates the priority information of the audio object based on the acquired data, and outputs the data including the audio signal and the metadata including the calculated priority information to a subsequent stage; a pass-through object selection unit that selects a pass-through object from among the plurality of audio objects based on the audio signal output from the priority calculation unit and the metadata including the calculated priority information; and the pass-through object selection unit outputs the audio signal of the pass-through object and the metadata including the calculated priority information to the encoding unit as they are, The decoding device a decoding unit that decodes the code string to obtain the audio signal of the audio object and the metadata including the calculated priority information. Information processing system.

Images