Information processing device and method, and program
By determining and applying loudness information for each position and orientation in a free viewpoint audio system, the system prevents clipping distortion, ensuring high-quality playback even when many objects are concentrated in areas with few speakers.
Patent Information
- Application Number
- PCT/JP2024/013177
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-22
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-28
AI Technical Summary
Free viewpoint audio systems experience clipping distortion when a large number of objects are concentrated in areas with few speakers, leading to output signal levels exceeding the recordable range of digital audio data, resulting in distorted playback.
The system determines loudness information for each position and orientation in a virtual space, allowing for level correction of audio data to prevent clipping distortion by generating and transmitting a bitstream containing this information, which is used for gain adjustment based on measured loudness values and target values.
This approach effectively suppresses clipping distortion by ensuring that the output signal levels remain within the recordable range, maintaining sound quality even in scenarios with a high concentration of objects and limited speakers.
Smart Images

Figure JP2024013177_28112024_PF_FP_ABST
Abstract
Description
Information processing device, method, and program
[0001] The present technology relates to an information processing device, method, and program, and in particular to an information processing device, method, and program that are capable of suppressing the occurrence of clipping distortion.
[0002] 2. Description of the Related Art Conventionally, a free viewpoint audio system is known that allows a listener to change their position and orientation within a virtual space.
[0003] Furthermore, as technologies relating to gain adjustment of audio data, a technology for determining the gain at the time of rendering based on the object position or the reference position (see, for example, Patent Document 1) and a technology for determining the loudness for each of multiple profiles have been proposed (see, for example, Patent Document 2).
[0004] JP 2021-193842 A JP 2022-166331 A
[0005] In a free viewpoint audio system, as the position and orientation of the listener changes in the virtual space, a large number of objects may end up concentrating in an area where there are few speakers arranged in the space.
[0006] In such cases, the level of the output signal from each speaker generated by the rendering process may exceed the range that can be recorded as digital audio data, resulting in clipping distortion. In other words, the reproduced sound may become clipped and distorted, resulting in a decrease in the quality of the reproduced sound.
[0007] The present technology has been made in consideration of such circumstances, and is intended to make it possible to suppress the occurrence of clipping distortion.
[0008] An information processing device according to a first aspect of the present technology includes an acquisition unit that acquires loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and a level correction unit that performs level correction on audio data of the object based on the loudness information for each of the plurality of positions or orientations.
[0009] An information processing method according to a first aspect of the present technology includes a step of acquiring loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and performing level correction on audio data of the object based on the loudness information for each of the plurality of positions or orientations.
[0010] In a first aspect of the present technology, loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed is obtained, and level correction is performed on the audio data of the object based on the loudness information for each of the plurality of positions or orientations.
[0011] An information processing device according to a second aspect of the present technology includes a generation unit that generates a bitstream including loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and a communication unit that transmits the bitstream.
[0012] In a second aspect of the present technology, a bitstream including loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed is generated, and the bitstream is transmitted.
[0013] An information processing device according to a third aspect of the present technology includes a correction unit that corrects gain information of the object, which is determined with respect to a control viewpoint, based on a measured loudness value, which is a measurement result of the loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener, and a predetermined target loudness value.
[0014] An information processing method according to a third aspect of the present technology includes a step of correcting gain information of the object, which is determined with respect to a control viewpoint, based on a measured loudness value, which is a measurement result of the loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener, and a predetermined target loudness value.
[0015] In a third aspect of the present technology, gain information of the object defined for a control viewpoint in space is corrected based on a measured loudness value, which is a measurement result of the loudness of audio data of content including sounds of one or more objects when the control viewpoint in space is set to the position of the listener, and a predetermined target loudness value.
[0016] An information processing device according to a fourth aspect of the present technology includes a control unit that generates configuration information including a measured loudness value, which is a measurement result of the loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener.
[0017] An information processing method according to a fourth aspect of the present technology includes a step of generating configuration information including a measured loudness value that is a result of measuring the loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener.
[0018] In a fourth aspect of the present technology, configuration information is generated that includes a measured loudness value that is a measurement result of the loudness of audio data of content that includes sounds of one or more objects when a control viewpoint in a space is set to the position of a listener.
[0019] An information processing device according to a fifth aspect of the present technology includes a generation unit that generates a bitstream including configuration information in which measured loudness values, which are measurement results of loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener, are stored, and a communication unit that transmits the bitstream.
[0020] In a fifth aspect of the present technology, a bitstream is generated that includes configuration information storing measured loudness values that are the results of measuring the loudness of audio data of content that includes sounds of one or more objects when a control viewpoint in a space is set to the position of a listener, and the bitstream is transmitted.
[0021] 1 is a diagram illustrating clip distortion of reproduced sound; FIG. 1 is a diagram illustrating region division in the horizontal direction; FIG. 1 is a diagram illustrating region division in the vertical direction; FIG. 2 is a diagram illustrating division points; FIG. 2 is a diagram illustrating an example of multi-angle loudness information; FIG. 3 is a diagram illustrating an example configuration of a server; FIG. 4 is a diagram illustrating an example configuration of a client; FIG. 4 is a flowchart illustrating bitstream transmission processing; FIG. 5 is a flowchart illustrating output signal generation processing; FIG. 6 is a flowchart illustrating attenuation coefficient calculation processing; FIG. 7 is a flowchart illustrating output signal generation processing; FIG. 8 is a diagram illustrating CVP selection; FIG. 9 is a flowchart illustrating output signal generation processing; FIG. 10 is a diagram illustrating interpolation processing; FIG. 11 is a flowchart illustrating output signal generation processing; FIG. 12 is a diagram illustrating loudness modes; FIG. 13 is a diagram illustrating measurement loudness modes; FIG. 14 is a diagram illustrating group modes; FIG. 15 is a diagram illustrating group modes; FIG. 16 is a diagram illustrating production loudness modes; FIG. 17 is a diagram illustrating processing flows on the production side and playback side; FIG. 18 is a diagram illustrating an example syntax of multi-loudness information; FIG. 19 is a diagram illustrating switching of configuration information; FIG. 20 is a diagram illustrating an example display screen; FIG. 21 is a diagram illustrating an example display screen; FIG. 22 is a diagram illustrating an example display screen; FIG. 23 is a diagram illustrating an example configuration of an information processing device; FIG. 24 is a flowchart illustrating configuration information generation processing. FIG. 1 is a diagram illustrating an example of the configuration of a server. FIG. 2 is a flowchart illustrating a bitstream transmission process. FIG. 3 is a diagram illustrating an example of the configuration of a client. FIG. 4 is a diagram illustrating an example of the functional configuration of a client. FIG. 5 is a flowchart illustrating an output audio data generation process. FIG. 6 is a flowchart illustrating a metadata generation process. FIG. 7 is a diagram illustrating an example of the configuration of a computer.
[0022] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.
[0023] First Embodiment About the Present Technology The present technology makes it possible to suppress the occurrence of clipping distortion by determining loudness information for each of a plurality of directions for each position in space.
[0024] In particular, according to the present technology, even in a situation where the number of actual speakers is small and a large number of objects are concentrated in an area where the number of speakers is small, it is possible to reduce the occurrence of clipping distortion.
[0025] For example, consider a free viewpoint audio system that plays back content consisting of sounds from multiple objects, as indicated by arrow Q11 in FIG.
[0026] In this case, assume that in the real space, many speakers are placed in front of the listener (user), i.e., on the upper side in the figure, and only a few speakers are placed behind the listener, i.e., on the lower side in the figure. Also, assume that in the real space, the user always listens to the sound of the content while facing forward.
[0027] In a virtual space (virtual space) where multiple objects that make up the content are placed, the listener can move his / her position and change the direction he / she is facing (listener orientation).
[0028] For example, in a virtual space, when the user faces the object side, that is, faces forward, as indicated by arrow A11, the sounds of multiple objects are output from multiple speakers, as indicated by arrow Q12.
[0029] In this case, although there are a large number of objects, there are also a large number of speakers responsible for outputting the sounds of these objects, so the load on each speaker is low.
[0030] In contrast, for example, in a virtual space, if the user is facing away from the objects, i.e., backwards, as indicated by arrow A12, the sounds of multiple objects will be output from a small number of speakers, as indicated by arrow Q13.
[0031] In this case, since there are a large number of objects but only a small number of speakers responsible for outputting the sounds of these objects, the load on each speaker becomes high.
[0032] In this way, in a free viewpoint audio system, as the listener's position in the virtual space moves or their listening direction (orientation) changes, a large number of objects may end up concentrating in an area with few real speakers, such as the rear direction in the real space, i.e., behind the listener.
[0033] This causes the level of the output signal from each speaker generated by the rendering process to exceed the range that can be recorded as digital audio data, such as PCM (Pulse Code Modulation) data, resulting in clipped and distorted playback sound.
[0034] Therefore, in this technology, by defining loudness information for each of a plurality of positions or directions, it is possible to reduce (suppress) clipping distortion of the reproduced sound for any viewpoint and listening direction.
[0035] Now, the present technology will be described below.
[0036] For example, in this technology, in a virtual space in which one or more objects constituting content are placed, a content creator pre-specifies (sets) multiple viewpoint positions as control viewpoints (hereinafter also referred to as CVPs). Note that the virtual space in which the objects are placed may be a two-dimensional space or a three-dimensional space, but the following description will be given assuming that the virtual space is a three-dimensional space.
[0037] The content creator specifies (sets) in advance as a CVP (control viewpoint) the position in the virtual space where the listener is desired to listen to the content when the content is played back, i.e., the viewpoint from which the sound of the content is desired to be heard. Note that the content may be, for example, audio content consisting of sound only, or video content consisting of video and accompanying audio.
[0038] For example, in order to enhance the artistic (musical) quality of content during the process of creating it, one may want to intentionally place objects in positions that are different from where they are visible, without being bound by the physical placement of the objects.
[0039] Therefore, this technology allows content creators to set up multiple CVPs in a virtual space and determine the placement position of objects for each CVP. In other words, even if the object is the same, the placement position of the object in the virtual space will differ for each CVP. This allows for the creation of highly artistic content.
[0040] Furthermore, when creating content, all CVPs existing in the three-dimensional virtual space are divided into regions horizontally and vertically based on the positions of the CVPs. For example, the region is divided horizontally first, and then vertically.
[0041] Specifically, suppose one CVP is placed at position P11 in virtual space, as indicated by arrow Q21 in Figure 2. In this case, suppose that horizontal region division is performed by placing multiple points as division points on the surface of a sphere centered on the CVP (position P11).
[0042] In this example, eight division points, including division point DV11, are located on a horizontal plane that includes position P11 and is also on the surface of the sphere. Here, each circle drawn on the surface of the sphere represents one division point. Also, the straight line that indicates the direction of the division point as viewed from the CVP (position P11) is referred to as the division line. In this example, the position of the intersection of division line L11 and the sphere is division point DV11.
[0043] When creating content, for example, the horizontal area as seen from the CVP is equally divided by a predetermined number of division lines, and division points are set at the intersections of each division line and the sphere.
[0044] Therefore, in this example, when the CVP (position P11) is viewed vertically, a total of eight division lines are provided at 45-degree intervals from position P11, which is the center of the sphere, as shown by arrow Q22, thereby providing eight division points. Note that, although an example in which the region is equally divided by the division lines will be described here, the region may also be divided unequally in the horizontal direction.
[0045] Furthermore, each division point, or more specifically, each horizontal division line, is assigned a horizontal division index j to identify that division line. In Figure 2, a number written near each division point indicates the value of the horizontal division index j. Therefore, for example, it can be seen that the horizontal division index j of the horizontal division line L11 at division point DV11 is set to "5."
[0046] For example, when playing back sound from content, if it is necessary to select one horizontal dividing line (division point) depending on the listener's orientation (listening direction), the dividing line that forms the smallest angle with the line L12 indicating the listener's orientation, as shown by arrow Q23, may be selected. In other words, the dividing line whose horizontal orientation is closest to the listener's orientation indicated by line L12 is selected. In this example, dividing line L11, which forms the smallest angle with line L12, is selected.
[0047] For example, if there are two dividing lines that form the smallest angle with the straight line L12, that is, if the direction between the directions of the two dividing lines is the listener's orientation, the dividing line that is closest to the listener's orientation at the immediately preceding time may be selected, taking into account changes in the listener's orientation over time.Otherwise, if some value corresponding to the dividing lines, such as loudness information, is used, proportional division may be performed between the two dividing lines.
[0048] In the following description, dividing lines for dividing areas in the horizontal direction, such as dividing line L11, will also be referred to as horizontal dividing lines.
[0049] After the horizontal division has been performed as described above, the next step is vertical division. For example, for each horizontal division line, the image is divided vertically from the direction of the horizontal division line as viewed from the CVP.
[0050] Specifically, for example, as indicated by arrow Q31 in Figure 3, three division points, including division point DV21, are provided on a vertical plane including position P11 and one horizontal division line, and on the surface of the sphere. Note that in Figure 3, parts corresponding to those in Figure 2 are given the same reference numerals, and their explanation will be omitted where appropriate.
[0051] In Figure 3, each circle drawn on the surface of the sphere represents one division point. The straight line indicating the direction of the division point as viewed from the CVP (position P11) is the division line. In this example, the intersection of division line L21 and the sphere is division point DV21.
[0052] When creating content, for example, the vertical area as seen from the CVP is divided by a predetermined number of vertical division lines, and division points are set at the intersections of the division lines and the sphere.
[0053] In this example, the direction of a given horizontal dividing line is set to a vertical angle, i.e., a vertical angle of 0 degrees, and dividing lines are provided in three directions where the vertical angle is 0 degrees, -90 degrees, and 90 degrees. In this case, the dividing line in the direction where the vertical angle is 0 degrees is equal to (the same as) one horizontal dividing line.
[0054] By dividing the area in the vertical direction in this way, three division points are provided in the example of FIG.
[0055] Furthermore, each division point, or more specifically, each vertical division line, is assigned a vertical division index k to identify that division line. In Figure 3, a number written near each division point indicates the value of the vertical division index k. Therefore, for example, it can be seen that the value of the vertical division index k of the vertical division line L21 at division point DV21 is set to "0."
[0056] For example, when playing back sound from content, if it is necessary to select a vertical dividing line (division point) depending on the listener's orientation (listening direction), the dividing line that forms the smallest angle with the line L22 indicating the listener's orientation, as shown by arrow Q32, may be selected. In other words, the dividing line whose vertical orientation is closest to the listener's orientation indicated by line L22 is selected. In this example, dividing line L21, which forms the smallest angle with line L22, is selected.
[0057] Hereinafter, dividing lines for dividing areas in the vertical direction, such as dividing line L21, will also be referred to as vertical dividing lines.
[0058] After dividing the region horizontally and vertically, division points are set corresponding to each division line. In other words, division points are set corresponding to multiple directions (orientations) relative to the CVP, i.e., multiple orientations that a listener in the CVP can take.
[0059] For example, when the horizontal region division shown in Fig. 2 and the vertical region division shown in Fig. 3 are performed, a total of 24 division points are provided as shown in Fig. 4. In Fig. 4, parts corresponding to those in Fig. 2 or 3 are given the same reference numerals, and their explanation will be omitted where appropriate.
[0060] In the example of Figure 4, a division point is set for each combination of horizontal division index j and vertical division index k for one CVP. In this example, the CVP is divided into 8 horizontally and 3 vertically, so a total of 24 division points are set.
[0061] In order to make it easier to understand the number of division points, the division points corresponding to the division lines with vertical angles of -90 degrees and 90 degrees are drawn at slightly shifted positions, but these division points are actually located in the same positions.
[0062] When creating content, loudness information loudnessInfo[i][j][k] is calculated in advance for each division point set for each CVP when a listener in the CVP faces the direction corresponding to that division point.
[0063] For example, loudness information loudnessInfo[i][j][k] set for a certain division point of a specific CVP is information about the loudness of the sound of the content that is played when the listener is facing the direction of the division point at the position of the specific CVP. In this case, the loudness information is calculated (determined) based on the audio data of all objects that make up the content, for example.
[0064] The loudness information loudnessInfo[i][j][k] is information used for gain control (level correction) of the audio data of an object. The elements i, j, and k of the array in the loudness information loudnessInfo[i][j][k] represent the CVP index i, horizontal division index j, and vertical division index k, respectively, which identify the CVP.
[0065] The loudness information can include information such as DRC (Dynamic Range Control) specified in the MPEG (Moving Picture Experts Group) standard, for example, ISO23003-4 Information technology MPEG audio technologies Part 4: Dynamic Range Control, and peak values used for gain control using a Peak Limiter.
[0066] That is, for example, the loudness information can include sample peak level values and true peak level values throughout the entire content, which are determined in advance by the content creator or the like.
[0067] The sample peak level value here refers to the peak value (maximum value) of the sample values in the entire PCM data as audio data after rendering processing, and the true peak level value refers to the peak value of the entire waveform of the audio signal after DA (Digital to Analog) conversion, obtained by performing rendering processing, DRC processing, etc. as appropriate.
[0068] For simplicity, the following description will be focused on the case where loudness information loudnessInfo[i][j][k] is a sample peak level value. Therefore, unless otherwise specified, loudness information will be considered to indicate a sample peak level value.
[0069] In the present technology, a plurality of the division points described above are set for each CVP, and loudness information is determined for the plurality of division points for each CVP. Note that one piece of loudness information may be determined for one CVP.
[0070] In other words, for one or more CVPs, loudness information is defined for one or more directions (orientations) relative to the CVP, i.e., for one or more possible orientations of a listener in the CVP. This can also be said to be defining loudness information for multiple possible positions or orientations of a listener in a virtual space. Here, loudness information is not defined for all possible positions and orientations of the listener, but for a discrete number of positions and directions among the positions to which the listener can move and the directions to which the listener can face.
[0071] For example, loudness information can be described as shown in Fig. 5. In this technology, for example, multi-angle loudness information including loudness information for each division point of each CVP is described in the bitstream in the format shown in Fig. 5.
[0072] In the example shown in Figure 5, NumOfControlViewpoints indicates the number of CVPs (CVP number), and the multi-angle loudness information stores the number of horizontal divisions numOfDivs_h[i] and the number of vertical divisions numOfDivs_v[i] for each CVP, i.e., for each CVP.
[0073] In addition, the multi-angle loudness information stores loudness information loudnessInfo[i][j][k] for each CVP, the number of division points being determined by the number of horizontal divisions numOfDivs_h[i] and the number of vertical divisions numOfDivs_v[i].
[0074] Therefore, in the example of FIG. 5, the multi-angle loudness information includes the number of horizontal divisions and the number of vertical divisions for each CVP, and loudness information for each division point for each CVP.
[0075] This technology suppresses clipping distortion during playback of content sound based on loudness information defined for each of a plurality of division points (directions) of each CVP.
[0076] First, as an example of a method for suppressing clipping distortion using loudness information, we will explain a method for suppressing clipping distortion by automatically and uniformly lowering the playback level of the output signal of the content supplied to all speakers.
[0077] This technique can be applied to free viewpoint audio content in which, for example, any position in a three-dimensional virtual space is set as the listener's position (viewpoint position), and the listener can freely move between a plurality of CVPs.
[0078] Specifically, in this method, on the content playback side (client side), the maximum value of the sample peak level values as loudness information is calculated as the maximum peak value maxPeak based on the loudness information at all division points of all CVPs.
[0079] In other words, all combinations of CVP index i, horizontal division index j, and vertical division index k are considered, and the largest value among the loudness information loudnessInfo[i][j][k] for each combination is determined as the maximum peak value maxPeak. This maximum peak value maxPeak is the loudness information in the direction (division point) where clipping distortion is greatest.
[0080] The maximum peak value maxPeak can be calculated, for example, by the following equation (1).
[0081]
[0082] Next, based on the maximum peak value maxPeak, the amount by which the maximum peak value maxPeak exceeds the maximum recordable value of the audio data is calculated as an excess value, and then, based on the excess value, an attenuation coefficient decayFac is calculated, which indicates the amount of attenuation required to adjust the peak value of the audio data (output signal) to the maximum recordable value.
[0083] If the output signals of all channels for all frames are uniformly attenuated by the amount of attenuation indicated by the attenuation coefficient decayFac through level correction (gain adjustment), clipping distortion should no longer occur when the sound of content based on the output signals is played back.
[0084] The maximum recordable value here refers to the maximum value (range) that the sound level based on audio data can take (maximum sound level) and can be recorded as digital audio data such as PCM data. For example, the maximum recordable value is 0 dB.
[0085] As a specific example, when the maximum recordable value is 0 [dB], the decay coefficient decayFac is calculated by the following formula (2): In formula (2), "^" represents exponentiation.
[0086]
[0087] The decay coefficient decayFac is calculated based on the maximum peak value maxPeak, i.e., the loudness information loudnessInfo[i][j][k], and then applied to the output signal for each speaker obtained by the rendering process. This attenuates the level of the playback sound based on the output signal supplied to each speaker to below the maximum recordable value, thereby suppressing clipping distortion.
[0088] The output signal is an audio signal generated by the rendering process and supplied to a speaker corresponding to each channel of the multi-channel configuration.
[0089] <Server Configuration Example> FIG. 6 is a diagram showing a configuration example of an embodiment of a server to which the present technology is applied.
[0090] The server 11 shown in FIG. 6 is an information processing device, and includes an acquisition unit 21, a bit stream encoder 22, and a communication unit 23.
[0091] The acquisition unit 21 acquires audio data (Object Audio) of each object constituting the content, system configuration information (Config Info), metadata of each object (Object Metadata), and loudness information (Loudness Info) for each division point of each CVP from inside the server 11 or from outside the server 11. Note that data other than the above-mentioned data may also be acquired.
[0092] The system configuration information is information about the entire content. For example, the system configuration information includes the number of objects that make up the content, the number of CVPs set in the virtual space, and CVP position information that indicates the absolute position of each CVP in the virtual space.
[0093] The object metadata includes object location information that indicates the location of the object for each CVP.
[0094] The object position information may be coordinate information that indicates the relative position of the object as seen from the CVP, expressed in polar coordinates, or it may be coordinate information that indicates the absolute position of the object in virtual space, defined for each CVP and expressed in absolute coordinates (Cartesian coordinates).
[0095] Furthermore, in addition to object position information, the object metadata may also include, for example, gain information of the object's audio data, priority information, and spread information indicating the extent of the object's spread.
[0096] The acquisition unit 21 supplies the acquired audio data (Object Audio), system configuration information (Config Info), metadata (Object Metadata), and loudness information (Loudness Info) to the bitstream encoder 22 .
[0097] The bit stream encoder 22 functions as a generating unit that generates a bit stream by appropriately encoding the audio data, system configuration information, metadata, and loudness information supplied from the acquisition unit 21 and supplies it to the communication unit 23.
[0098] The bitstream contains audio data, system configuration information, metadata, and multi-angle loudness information, which includes the number of horizontal and vertical divisions for each CVP, and loudness information for each division point for each CVP.
[0099] The system configuration information and multi-angle loudness information may be generated for each frame of the content, for each section consisting of multiple frames, or for the entire content. Furthermore, the system configuration information and multi-angle loudness information may be transmitted at a timing different from that of the audio data of the object.
[0100] The communication unit 23 transmits the bit stream supplied from the bit stream encoder 22 to a client, which is an information processing device on the content playback side.
[0101] <Configuration Example of Client> FIG. 7 is a diagram showing a configuration example of an embodiment of a client to which the present technology is applied.
[0102] 7 is an information processing device that receives a bitstream transmitted by the server 11 and plays back the content. The client 51 is, for example, a personal computer, a tablet terminal, or a smartphone.
[0103] The client 51 has a communication unit 61, a bitstream decoder 62, a metadecoder 63, a rendering processing unit 64, a loudness information processing unit 65, and a DRC processing unit 66. The client 51 is also connected to a speaker 71, which is a speaker system with a multi-channel configuration.
[0104] The communication unit 61 receives the bit stream transmitted from the server 11 and supplies it to the bit stream decoder 62. The communication unit 61 functions as an acquisition unit that acquires loudness information by receiving the bit stream.
[0105] The bitstream decoder 62 functions as a decoding unit that decodes the bitstream supplied from the communication unit 61, more specifically, the encoded audio data and the like included in the bitstream. By decoding and the like in the bitstream decoder 62, audio data, system configuration information, metadata, and multi-angle loudness information are extracted from the bitstream.
[0106] The bitstream decoder 62 supplies audio data to a rendering processing unit 64, and also supplies system configuration information to a metadecoder 63 and a loudness information processing unit 65. The bitstream decoder 62 also supplies metadata to the metadecoder 63, and also supplies multi-angle loudness information to the loudness information processing unit 65.
[0107] The meta decoder 63 and the loudness information processing unit 65 are supplied with listener position information that indicates the absolute position of the listener in the three-dimensional virtual space in which the objects are arranged.
[0108] In addition, listener orientation information indicating the orientation of the listener in the three-dimensional virtual space is also supplied to the meta decoder 63 and the loudness information processing unit 65 as appropriate. For example, the listener orientation information consists of a yaw angle (horizontal angle) indicating the horizontal orientation of the listener and a pitch angle (vertical angle) indicating the vertical orientation of the listener. Note that the listener orientation information may also include a roll angle indicating the rotation angle of the listener.
[0109] The meta decoder 63 generates listener reference object position information based on the supplied listener position information and the metadata and system configuration information from the bit stream decoder 62 , and supplies the generated information to the rendering processing unit 64 .
[0110] For example, the listener-reference object position information is information that indicates the relative position of an object as seen from the listener, expressed using coordinates (polar coordinates) in a polar coordinate system with the listener's position in virtual space as the reference (origin).
[0111] For example, if the object position information included in the metadata is coordinate information indicating the relative position of the object as seen from the CVP, the meta decoder 63 calculates position information indicating the absolute position of the object in virtual space based on the CVP position information and object position information included in the system configuration information, and generates (calculates) listener-reference object position information based on the calculated position information and the listener position information.
[0112] Also, for example, if the object position information included in the metadata is coordinate information indicating the absolute position of the object in virtual space, the meta decoder 63 generates listener-reference object position information based on the object position information and the listener position information.
[0113] The rendering processing unit 64 performs rendering processing based on the audio data of the object supplied from the bitstream decoder 62 and the listener reference object position information supplied from the meta decoder 63, and generates an output signal for each channel.
[0114] The rendering processing unit 64 generates an output signal by performing rendering processing in a polar coordinate system defined by MPEG-H, such as VBAP (Vector Based Amplitude Panning). Note that the rendering processing is not limited to VBAP and may be any other processing. For example, BRIR (Binaural Room Impulse Response), HRTF (Head Related Transfer Function), HOA (Higher Order Ambisonics), etc. may be used for the rendering processing.
[0115] The output signal of each channel is audio data (audio signal) supplied to a speaker corresponding to each channel constituting a speaker system as speaker 71. The sound of the content including the sounds of all objects is reproduced in speaker 71 by the output signal of each channel. At this time, the sound (sound image) of each object is localized at a position in the virtual space indicated by the object position information.
[0116] The loudness information processing unit 65 performs processing related to the loudness information using the supplied listener position information, the multi-angle loudness information from the bitstream decoder 62 and the system configuration information as appropriate, and supplies the processing results to the DRC processing unit 66.
[0117] For example, the loudness information processing unit 65 calculates the decay coefficient decayFac described above based on the multi-angle loudness information, that is, the loudness information of all division points of all CVPs, and supplies it to the DRC processing unit 66 .
[0118] The DRC processing unit 66 performs level correction (gain adjustment) on the output signal based on the processing result related to the loudness information supplied from the loudness information processing unit 65 and the output signal supplied from the rendering processing unit 64, and supplies the level-corrected output signal as a final output signal to the speaker 71. In other words, the DRC processing unit 66 functions as a level correction unit (gain adjustment unit) that performs level correction on the output signal based on loudness information for each of the multiple division points of each CVP.
[0119] For example, the DRC processing unit 66 performs level correction of the output signal based on the decay coefficient decayFac supplied from the loudness information processing unit 65, and, if necessary, performs level correction by DRC processing on the output signal after level correction based on the decay coefficient decayFac. More specifically, the DRC processing unit 66 performs DA conversion on the output signal, which is a digital signal, and supplies the resulting output signal, which is an analog signal, to the speaker 71.
[0120] The speaker 71 reproduces the sound of the content based on the output signal supplied from the DRC processing unit 66 .
[0121] <Description of Bitstream Transmission Process> Next, the operations of the server 11 and the client 51 will be described.
[0122] First, the bitstream transmission process by the server 11 will be described with reference to the flowchart of FIG.
[0123] In step S11 , the acquisition unit 21 acquires information necessary for generating a bitstream and supplies it to the bitstream encoder 22 .
[0124] For example, the acquisition unit 21 acquires audio data of each object, system configuration information, metadata of each object, the number of horizontal divisions and the number of vertical divisions of each CVP, loudness information for each division point of each CVP, and the like.
[0125] In step S12 , the bitstream encoder 22 generates a bitstream by encoding and multiplexing the audio data, system configuration information, metadata, and loudness information supplied from the acquisition unit 21 as appropriate, and supplies the generated bitstream to the communication unit 23 .
[0126] In step S13, the communication unit 23 transmits the bit stream supplied from the bit stream encoder 22 to the client 51, and the bit stream transmission process ends.
[0127] In this way, the server 11 generates a bitstream containing loudness information for each CVP for each of a plurality of directions seen from the CVP, i.e., for each of a plurality of division points, and transmits it to the client 51. This allows the client 51 receiving the bitstream to suppress clipping distortion during content playback based on this loudness information.
[0128] <Description of Output Signal Generation Processing> Next, the output signal generation processing by the client 51 will be described with reference to the flowchart of Fig. 9. This output signal generation processing starts when the processing of step S13 in Fig. 8 is performed.
[0129] In step S41, the communication unit 61 receives the bit stream transmitted from the server 11 in step S13 of FIG.
[0130] In step S42, the bitstream decoder 62 decodes the encoded audio data and the like contained in the bitstream supplied from the communication unit 61, and extracts various pieces of information contained in the bitstream, thereby extracting audio data, system configuration information, metadata, and multi-angle loudness information from the bitstream.
[0131] The bitstream decoder 62 supplies the audio data to a rendering processing unit 64 , supplies the system configuration information and metadata to a meta decoder 63 , and supplies the system configuration information and multi-angle loudness information to a loudness information processing unit 65 .
[0132] In step S43, the loudness information processing unit 65 performs an attenuation coefficient calculation process to calculate an attenuation coefficient decayFac based on the loudness information of all division points of all CVPs and the maximum recordable value, and supplies the calculated attenuation coefficient decayFac to the DRC processing unit 66. Details of the attenuation coefficient calculation process will be described later.
[0133] In step S46 (described later), the same decay coefficient decayFac is used for all channels and all frames, so the process of step S43 is performed only once. In contrast, the processes of steps S44 to S46 (described later) are performed for each frame of the content (audio data). In particular, step S46 is performed for each channel corresponding to the speaker 71 for each frame.
[0134] In step S44 , the meta decoder 63 calculates (generates) listener reference object position information based on the supplied listener position information and the metadata and system configuration information from the bit stream decoder 62 , and supplies the listener reference object position information to the rendering processing unit 64 .
[0135] In step S45, the rendering processing unit 64 performs rendering processing such as VBAP based on the audio data of the object supplied from the bitstream decoder 62 and the listener reference object position information supplied from the meta decoder 63. For example, in the rendering processing, an output signal for each channel for reproducing the sound of each object is generated for each object. Then, the output signals of the same channel obtained for each object are added together to form a final output signal for each channel. The rendering processing unit 64 supplies the output signal for each channel obtained by the rendering processing to the DRC processing unit 66.
[0136] In step S 46 , the DRC processing unit 66 performs DRC processing on the output signal supplied from the rendering processing unit 64 .
[0137] For example, the DRC processing unit 66 performs level correction (gain adjustment) on the output signal of each channel based on the attenuation coefficient decayFac supplied from the loudness information processing unit 65 .
[0138] Specifically, for example, the output signal before level correction, i.e., the signal output by the rendering processing unit 64, is represented as render_out[fr][ch], and the output signal after level correction is represented as out[fr][ch], where fr and ch are indexes indicating the frame and channel, respectively.
[0139] In this case, the DRC processing unit 66 generates the level-corrected output signal out[fr][ch] by calculating the following equation (3) for each channel for each frame. That is, in equation (3), the output signal render_out[fr][ch] is multiplied by the attenuation coefficient decayFac to perform level correction.
[0140]
[0141] In addition, the DRC processing unit 66 further performs DRC processing on the output signal after level correction, and also performs DA conversion on the output signal obtained by the DRC processing, and supplies the resulting analog output signal to the speaker 71.
[0142] The speaker 71 outputs (plays) the sound of the content based on the output signal supplied from the DRC processing unit 66. When the sound of the content is played back by the speaker 71, the output signal generation process ends.
[0143] In the DRC processing section 66, as long as level correction based on the decay coefficient decayFac is performed, there is no need to perform DRC processing.
[0144] In this way, the client 51 calculates the decay coefficient decayFac based on the loudness information and corrects the level of the output signal based on the decay coefficient decayFac. In this way, the level of the reproduced sound based on the output signal is attenuated to a value equal to or less than the maximum recordable value, thereby suppressing the occurrence of clipping distortion.
[0145] <Description of Damping Coefficient Calculation Process> The damping coefficient calculation process corresponding to the process in step S43 in FIG. 9 will be described with reference to the flowchart in FIG.
[0146] In step S71, the loudness information processing unit 65 obtains the number of horizontal divisions and the number of vertical divisions for each CVP from the multi-angle loudness information supplied from the bit stream decoder 62.
[0147] For example, the loudness information processing unit 65 reads the number of horizontal divisions numOfDivs_h[i] and the number of vertical divisions numOfDivs_v[i] for each CVP index i from the multi-angle loudness information shown in Fig. 5. This specifies the number of divisions in the horizontal and vertical directions for each CVP.
[0148] In step S72, the loudness information processing unit 65 sets the maximum peak value cur_peak to 0.0. This maximum peak value cur_peak corresponds to the above-mentioned maximum peak value maxPeak, and is currently set to a temporary value of "0.0."
[0149] In step S73, the loudness information processing unit 65 selects one of the multiple CVPs set in the virtual space as the CVP to be processed, based on the system configuration information supplied from the bit stream decoder 62.
[0150] In step S74, the loudness information processing unit 65 selects one of the division points determined for the CVP to be processed based on the multi-angle loudness information. For example, a division point determined by a combination of a horizontal division index j and a vertical division index k is selected as the division point to be processed.
[0151] The loudness information processing unit 65 reads out loudness information loudnessInfo[i][j][k] determined for the division point of the CVP to be processed from the multi-angle loudness information. In this case, the loudness information loudnessInfo[i][j][k] is particularly set as a sample peak level value.
[0152] In step S75, the loudness information processing unit 65 determines whether the value of the read loudness information loudnessInfo[i][j][k] of the division point to be processed is greater than the maximum peak value cur_peak.
[0153] If it is determined in step S75 that the value of the loudness information is greater than the maximum peak value cur_peak, then the process of step S76 is carried out.
[0154] In step S76, the loudness information processing unit 65 updates the maximum peak value cur_peak to the value of the loudness information loudnessInfo[i][j][k] of the division point being processed. As a result of this update, the value of the maximum peak value cur_peak becomes the value of the loudness information with the largest value among the loudness information of the division points being processed so far.
[0155] After the process of step S76 is performed and the maximum peak value cur_peak is updated, the process then proceeds to step S77.
[0156] On the other hand, if it is determined in step S75 that the loudness information value is not greater than the maximum peak value cur_peak, the processing of step S76 is not performed, i.e., the maximum peak value cur_peak is not updated, and then the processing proceeds to step S77.
[0157] If the process of step S76 is performed or if it is determined in step S75 that the loudness information value is not greater than the maximum peak value cur_peak, the process of step S77 is performed.
[0158] In step S77, the loudness information processing unit 65 determines whether or not all division points of the CVP to be processed have been processed, i.e., whether or not all division points of the CVP to be processed have been processed.
[0159] If it is determined in step S77 that not all division points have been processed, i.e., if it is determined that not all division points have yet been processed, the process returns to step S74, and the above-described process is repeated. In this case, a new division point that has not yet been processed is selected as the processing target, and the above-described process is performed.
[0160] On the other hand, if it is determined in step S77 that all division points have been processed, the loudness information processing unit 65 determines in step S78 whether or not all CVPs have been processed, i.e., whether or not all CVPs have been processed.
[0161] If it is determined in step S78 that not all CVPs have been processed, that is, if it is determined that not all CVPs have yet been processed, the process returns to step S73 and the above-described process is repeated. In this case, a new CVP that has not yet been processed is selected as the processing target, and the above-described process is performed.
[0162] On the other hand, if it is determined in step S78 that all CVPs have been processed, then the process proceeds to step S79.
[0163] The above processing of steps S73 to S78 can be said to be processing for performing the calculation of the above-mentioned equation (1). The maximum peak value cur_peak is the loudness information with the largest value among the loudness information of all division points of all CVPs.
[0164] In step S79, the loudness information processing unit 65 determines whether the final maximum peak value cur_peak is greater than a predetermined maximum recordable value, which is set to, for example, 0.0 dB.
[0165] If it is determined in step S79 that the maximum peak value cur_peak is greater than the maximum recordable value, the loudness information processing unit 65 calculates the decay coefficient decayFac based on the maximum peak value cur_peak in step S80.
[0166] Specifically, the loudness information processing unit 65 obtains a gain gain_db based on the maximum peak value cur_peak as shown in the following equation (4), and calculates the attenuation coefficient decayFac by calculating the following equation (5) based on the obtained gain gain_db. In these equations (4) and (5), the same calculation as in the above-mentioned equation (2) is performed. That is, the calculations in equations (4) and (5) are the same as the calculation in equation (2).
[0167]
[0168]
[0169] On the other hand, if it is determined in step S79 that the maximum peak value cur_peak is equal to or less than the maximum recordable value, in step S81 the loudness information processing unit 65 sets the decay coefficient decayFac to "1.0" regardless of the value of the maximum peak value cur_peak. That is, the decay coefficient decayFac=1.0.
[0170] This is because clipping distortion does not occur when the content is played back if the maximum peak value cur_peak is equal to or less than the maximum recordable value. In such a case, the DRC processing unit 66 at the subsequent stage does not actually perform level correction for clipping distortion.
[0171] After the processing of step S80 or step S81 is performed, the loudness information processing unit 65 supplies the calculated attenuation coefficient decayFac to the DRC processing unit 66, and the attenuation coefficient calculation processing ends. When the attenuation coefficient calculation processing ends, step S43 in Fig. 9 has been performed, and the processing then proceeds to step S44 in Fig. 9.
[0172] In this way, the client 51 calculates the decay coefficient decayFac based on the loudness information at all division points of all CVPs. By using such decay coefficient decayFac, it is possible to suppress the occurrence of clipping distortion by level correction.
[0173] Second Embodiment Explanation of Output Signal Generation Processing In the first embodiment described above, the entire output signals of multiple channels are uniformly attenuated to match the maximum recordable value, which may result in the playback sound of the content becoming too quiet in some cases.
[0174] Therefore, by using adaptive level correction (gain adjustment) such as DRC processing standardized by MPEG, for example, it is possible to suppress the occurrence of clipping distortion without lowering the playback level of the entire output signal.
[0175] In the second embodiment, an example of suppressing clipping distortion by DRC processing using loudness information will be described. The technique described in the second embodiment is applicable to multi-view audio content in which, for example, the position of any one of a plurality of CVPs can be selected as the listener's position in a virtual space, that is, discrete movement between a plurality of CVPs is possible.
[0176] In the method described in the second embodiment, for example, in the CVP selected as the current listener position, loudness information of the division point closest to the listener's orientation is selected. That is, among the directions indicated by each combination of horizontal and vertical division lines, loudness information determined for the combination closest to the listener's orientation is selected. Then, based on the selected loudness information, the level of the output signal is corrected by DRC processing or the like.
[0177] When the occurrence of clipping distortion is suppressed by DRC processing using loudness information, the client 51 performs, for example, an output signal generation process shown in FIG.
[0178] The output signal generation process by the client 51 will be described below with reference to the flowchart of Fig. 11. Note that the processes of steps S111 and S112 are similar to the processes of steps S41 and S42 in Fig. 9, and therefore description thereof will be omitted.
[0179] In step S113, the loudness information processing unit 65 obtains the number of horizontal divisions and the number of vertical divisions for each CVP from the multi-angle loudness information supplied from the bit stream decoder 62.
[0180] In step S113, the same process as in step S71 in FIG. 10 is performed, and the number of horizontal divisions numOfDivs_h[i] and the number of vertical divisions numOfDivs_v[i] are read out for each CVP index i.
[0181] In step S114, the loudness information processing unit 65 acquires a CVP index i=cur_cvp indicating the CVP selected by the listener based on a signal supplied by the listener's input operation, etc. The loudness information processing unit 65 may also identify the CVP index i=cur_cvp from the supplied listener position information.
[0182] Hereinafter, the CVP index i indicating the CVP selected by the listener will also be referred to as the CVP index cur_cvp where appropriate.
[0183] In step S115, the loudness information processing unit 65 acquires listener orientation information supplied in response to an input operation by the listener, etc. For example, the listener orientation information includes a yaw angle indicating the horizontal orientation of the listener at the current time in the virtual space and a pitch angle indicating the vertical orientation.
[0184] In step S116, the loudness information processing unit 65 calculates (identifies) the horizontal division index j indicating the horizontal division line closest to the listener's horizontal orientation, among the horizontal division indexes j of the CVP indicated by the CVP index cur_cvp, based on the multi-angle loudness information and the listener orientation information. In this case, for example, in the example shown in Figure 2, the horizontal division index j = 5 of the division line L11 is obtained as the identification result (calculation result). It is assumed that the direction of the horizontal division line can be identified by the value of the horizontal division index j.
[0185] In the following, the value of the horizontal split index j calculated in step S116 is assumed to be "sel_hidx", and will also be referred to as the horizontal split index sel_hidx where appropriate.
[0186] In step S117, the loudness information processing unit 65 calculates (specifies) the vertical division index k indicating the vertical division line closest to the vertical orientation of the listener, among the vertical division indexes k of the CVP indicated by the CVP index cur_cvp, based on the multi-angle loudness information and the listener orientation information. In this case, for example, in the example shown in Figure 3, the vertical division index k = 0 of the division line L21 is obtained as the specified result. Note that the direction of the vertical division line can be specified by the value of the vertical division index k.
[0187] In the following, the value of the vertical split index k calculated in step S117 is assumed to be "sel_vidx", and will also be referred to as the vertical split index sel_vidx where appropriate.
[0188] In step S118 , the loudness information processing unit 65 selects loudness information according to the orientation of the listener, reads the selected loudness information from the multi-angle loudness information, and supplies the selected loudness information to the DRC processing unit 66 .
[0189] Specifically, the loudness information processing unit 65 selects loudness information loudnessInfo[cur_cvp][sel_hidx][sel_vidx] determined by a combination of the CVP index cur_cvp, the horizontal division index sel_hidx, and the vertical division index sel_vidx.
[0190] The loudness information selected in this manner is the loudness information associated with the division point located in the direction closest to the listener's orientation in the CVP selected by the listener, i.e., the loudness information for the direction closest to the orientation indicated by the listener orientation information among the loudness information for each of a plurality of orientations based on the CVP.
[0191] After the loudness information is selected in step S118, steps S119 and S120 are performed. These steps are similar to the steps S44 and S45 in FIG. 9, and therefore will not be described further.
[0192] In step S121 , the DRC processing unit 66 performs DRC processing on the output signal of each channel supplied from the rendering processing unit 64 based on the loudness information supplied from the loudness information processing unit 65 .
[0193] That is, the DRC processing unit 66 performs DRC processing and DA conversion based on the sample peak level value or true peak level value as loudness information to obtain an analog output signal that has been subjected to level correction (gain adjustment) based on the loudness information. That is, the level of the output signal is corrected by DRC processing or the like based on the loudness information. The DRC processing unit 66 supplies the output signal obtained in this manner to the speaker 71, which reproduces the sound of the content.
[0194] In step S122, the client 51 determines whether or not all frames of the content (audio data) have been processed. For example, in step S122, if the output signals of all frames have been supplied to the speaker 71 and played back, it is determined that all frames have been processed.
[0195] If it is determined in step S122 that the processing has not yet been performed on all frames, the process then returns to step S114, and the above-described processing is repeated for a new frame that has not yet been processed.
[0196] On the other hand, if it is determined in step S122 that processing has been performed on all frames, each unit of the client 51 ends the processing it is currently performing, and the output signal generation processing is thereby terminated.
[0197] In this way, the client 51 selects appropriate loudness information for the position and orientation of the listener, and performs level correction based on that loudness information. This makes it possible to prevent clipping distortion. Furthermore, by implementing level correction based on loudness information in DRC processing, etc., it is possible to prevent the playback volume of content from becoming too quiet.
[0198] Third Embodiment Selection of Loudness Information As in the second embodiment, even for free viewpoint audio content in which the listener can move freely within a virtual space by setting any position, including a position other than the CVP, as the listener's position (viewpoint position), it is possible to prevent the playback volume of the content from becoming too quiet.
[0199] In such a case, for example, based on the listener position information and listener orientation information, the loudness information of the division point located in the direction closest to the listener orientation, which is determined for the CVP closest to the listener's position, is selected, and DRC processing etc. is performed based on the selected loudness information.
[0200] For example, assume that CVP1, CVP2, and a listener exist in a three-dimensional virtual space as shown in FIG. 12, and the orientation of the listener is in the direction indicated by arrow L31.
[0201] Figure 12 shows the virtual space as seen from above. In Figure 12, each circle on a circle centered on a CVP represents a division point, and the arrows starting from the CVP position and ending at the division point represent division lines corresponding to each division point. Furthermore, the numbers written near each division point indicate the value of the horizontal division index j.
[0202] In this case, first, the CVP closest to the listener's position is selected from CVP1 and CVP2. In this case, CVP2 is selected because the distance b from the listener to CVP2 is shorter than the distance a from the listener to CVP1.
[0203] Next, the division point arranged in the direction closest to the listener's orientation is selected from among the multiple division points of CVP2. More specifically, the combination of horizontal and vertical division lines in the direction closest to the listener's orientation is selected, and the division point corresponding to the selected combination is selected. However, for simplicity, only the selection of the horizontal division line will be described here.
[0204] Arrow L32, which originates from CVP2, points in the same direction as the orientation of the listener indicated by arrow L31. Therefore, in this example, of the dividing lines (horizontal dividing lines) of CVP2, dividing line L33 indicating the direction closest to the direction indicated by arrow L32 is selected, and division point DV31 corresponding to dividing line L33 is selected. In other words, dividing line L33 that forms the smallest angle with the direction indicated by arrow L32 is selected, and division point DV31 corresponding to dividing line L33 is selected.
[0205] Therefore, in this example, DRC processing etc. is performed based on the loudness information determined for the division point DV31 of CVP2, thereby achieving level correction of the output signal.
[0206] In the third and second embodiments, the CVP closest to the listener's position is selected, but it is also possible that the distances between multiple CVPs and the listener's position are equal, i.e., the distance ratios are the same. In such a case, for example, listener position information used earlier in time, more specifically, immediately before the current time, may be used to select the CVP closest to the position indicated by that listener position information.
[0207] Alternatively, priority information indicating the priority of each CVP may be set in advance, and the CVP with the highest priority may be selected from among a plurality of CVPs closest to the listener.
[0208] <Description of Output Signal Generation Process> As described with reference to FIG. 12, when loudness information is selected based on the position and orientation of the listener, the client 51 performs output signal generation process shown in FIG. 13, for example.
[0209] The output signal generation process by the client 51 will be described below with reference to the flowchart of Fig. 13. Note that the processes of steps S151 to S153 are similar to the processes of steps S111 to S113 in Fig. 11, and therefore description thereof will be omitted.
[0210] In step S154, the loudness information processing unit 65 identifies the CVP index i=near_cvp indicating the CVP closest to the listener position in the virtual space based on the supplied listener position information and the system configuration information from the bitstream decoder 62, and acquires the CVP index. That is, the CVP closest to the listener position in the virtual space is selected.
[0211] Since the system configuration information includes CVP position information that indicates the absolute position of each CVP in the virtual space, it is possible to identify the CVP that is closest to the listener's position by calculation using the CVP position information and the listener position information. Here, we will continue the explanation assuming that there is one CVP closest to the listener's position.
[0212] Furthermore, hereinafter, the CVP index i indicating the CVP closest to the listener will also be referred to as the CVP index near_cvp where appropriate.
[0213] Once the CVP index near_cvp is acquired, the processes of steps S155 to S157 are performed. However, since these processes are similar to the processes of steps S115 to S117 in FIG. 11, a description thereof will be omitted.
[0214] However, in steps S156 and S157, the CVP indicated by the CVP index near_cvp is targeted, and the horizontal division index sel_hidx and vertical division index sel_vidx are identified for that CVP.
[0215] In step S158 , the loudness information processing unit 65 selects loudness information according to the position and orientation of the listener, reads the selected loudness information from the multi-angle loudness information, and supplies the selected loudness information to the DRC processing unit 66 .
[0216] Specifically, the loudness information processing unit 65 selects loudness information loudnessInfo[near_cvp][sel_hidx][sel_vidx] determined by a combination of the CVP index near_cvp, the horizontal division index sel_hidx, and the vertical division index sel_vidx.
[0217] The loudness information selected in this way is the loudness information associated with the division point in the CVP closest to the listener, which is arranged (located) in the direction closest to the listener's orientation with respect to that CVP. That is, from among the loudness information for each of a plurality of orientations with respect to the CVP, the loudness information for the orientation closest to the orientation indicated by the listener orientation information is selected.
[0218] Once the loudness information is selected in this manner, the processing of steps S159 to S162 is performed, and the output signal generation processing is completed. However, since these processing steps are the same as the processing steps S119 to S122 in FIG. 11, their description will be omitted.
[0219] In this way, the client 51 selects appropriate loudness information for the position and orientation of the listener, and performs level correction based on that loudness information. This makes it possible to prevent clipping distortion. Furthermore, by implementing level correction based on loudness information in DRC processing, etc., it is possible to prevent the playback volume of content from becoming too quiet.
[0220] <Fourth Embodiment> <Regarding Selection of Loudness Information> In free viewpoint audio content in which a listener can freely move to any position in a virtual space, including positions other than the CVP, it is possible to prevent the playback volume of the content from becoming too quiet by a method other than the method described in the third embodiment. In the fourth embodiment, such another method will be described.
[0221] In such a case, all CVPs are considered, and the loudness information of the division point located in the direction closest to the listener's orientation is selected for each CVP, i.e., the loudness information determined by the combination of horizontal division index j and vertical division index k corresponding to the listener's orientation is selected.
[0222] Then, based on the loudness information of each selected CVP, loudness information calculated by interpolating using the inverse ratio of the distance from the current listener position to each CVP is used to perform level correction of the output signal using DRC processing, etc.
[0223] For example, assume that CVP1, CVP2, CVP3, and a listener exist in a three-dimensional virtual space as shown in Fig. 14, and the orientation of the listener is in the direction indicated by arrow L31. Note that in Fig. 14, parts corresponding to those in Fig. 12 are given the same reference numerals, and their explanation will be omitted as appropriate.
[0224] 14, arrow L41 starting from CVP1 indicates the same direction as the orientation of the listener indicated by arrow L31. Therefore, of the division points of CVP1, the division point arranged in the direction closest to the direction indicated by arrow L41 (the orientation of the listener) is division point DV41.
[0225] Similarly, arrow L42 starting from CVP3 indicates the same direction as the orientation of the listener indicated by arrow L31, and of the division points of CVP3, the division point located closest to the direction indicated by arrow L42 (the orientation of the listener) is division point DV42.
[0226] In the example of FIG. 14, the position F of the listener indicated by the listener position information, that is, the coordinates of point F in the virtual space are (xf, yf, zf).
[0227] Furthermore, the positions of CVP1, CVP2, and CVP3 in the virtual space are assumed to be position A, position B, and position C, respectively, and the coordinates of positions A, B, and C indicated by the CVP position information are assumed to be (xa, ya, za), (xb, yb, zb), and (xc, yc, zc).
[0228] Furthermore, suppose that the distance AF from position A of CVP1 to listener position F is distance AF = a, the distance BF from position B of CVP2 to listener position F is distance BF = b, and the distance CF from position C of CVP3 to listener position F is distance CF = c.
[0229] In this case, the reciprocal ratios of the distances AF, BF, and CF are calculated, and interpolation processing is performed based on these reciprocal ratios and the loudness information of the division point DV41 of CVP1, the loudness information of the division point DV31 of CVP2, and the loudness information of the division point DV42 of CVP3.
[0230] By this interpolation process, loudness information corresponding to the listener's position F and the listener's orientation indicated by the arrow L31 is obtained, and the level of the output signal is corrected by DRC processing or the like based on the loudness information.
[0231] Although the case where all CVPs in the virtual space are targeted, division points (loudness information) are selected for each CVP, and interpolation processing is performed will be described here, it is also possible to target only a portion of a plurality of CVPs in the vicinity of the listener, rather than targeting all CVPs.Furthermore, when targeting only a portion of a plurality of CVPs in the vicinity of the listener, the number of target CVPs may be arbitrarily set by the user (listener), or may be dynamically changed depending on the client's resources, remaining battery power, transmission bandwidth, etc.
[0232] The interpolation process for obtaining loudness information will be described in detail below.
[0233] As shown in Figure 14, when CVP1, CVP2, and CVP3 are located at three positions A, B, and C in the virtual space, respectively, and the listener is at position F, the ratio (distance ratio) of distance AF = a, distance BF = b, and distance CF = c is as shown in the following equation (6).
[0234]
[0235] Furthermore, if the reciprocal of the distance ratio is taken as the degree of dependence (contribution) of the CVP on the loudness information to be calculated, the dependence ratio, which is the ratio of the degrees of dependence of each CVP, is as shown in the following equation (7).
[0236]
[0237] In equation (7), dp(AF), dp(BF), and dp(CF) represent the degree of dependence on CVP1, the degree of dependence on CVP2, and the degree of dependence on CVP3, respectively.
[0238] If the normalized dependencies dp(AF) to dp(CF) are defined as dependencies Cbr(AF), Cbr(BF), and Cbr(CF), respectively, the ratio of the normalized dependencies of each CVP, i.e., the dependency ratio, is given by the following formula (8): In formula (8), "^" represents exponentiation, and "sqrt" represents the square root.
[0239]
[0240] Interpolation processing is performed based on the dependency Cbr(AF) to dependency Cbr(CF) based on the reciprocal of the distance obtained in this manner and the loudness information corresponding to the listener orientation of CVP1 to CVP3, and loudness information corresponding to the listener's position and orientation is obtained.
[0241] Specifically, for example, when calculating a sample peak level value EstSamplePeakLevel as loudness information corresponding to the position and orientation of the listener, the calculation shown in the following equation (9) is performed.
[0242]
[0243] In equation (9), SamplePeakLevel[0][hor1][vir1] indicates the sample peak level value as loudness information of division point DV41 selected for CVP1 in FIG.
[0244] Therefore, in this example, division point DV41 is a division point specified by the combination of indexes: CVP index i = 0, horizontal division index j = hor1, and vertical division index k = vir1. In this case, the horizontal division index j of the horizontal division line indicating the direction closest to the horizontal orientation of the listener is "hor1 = 7," and the vertical division index k of the vertical division line indicating the direction closest to the vertical orientation of the listener is "vir1."
[0245] Similarly, in equation (9), SamplePeakLevel[1][hor2][vir2] indicates the sample peak level value as loudness information of division point DV31 selected for CVP2 in Fig. 14. SamplePeakLevel[2][hor3][vir3] indicates the sample peak level value as loudness information of division point DV42 selected for CVP3 in Fig. 14.
[0246] Furthermore, for example, when calculating a true peak level value EstTruePeakLevel as loudness information corresponding to the position and orientation of the listener, the calculation shown in the following equation (10) is performed.
[0247]
[0248] In equation (10), TruePeakLevel[0][hor1][vir1] indicates the true peak level value as loudness information of division point DV41 selected for CVP1 in FIG.
[0249] In equation (10), as in equation (9), division point DV41 is a division point identified by the combination of the indexes: CVP index i = 0, horizontal division index j = hor1 = 7, and vertical division index k = vir1.
[0250] In addition, in equation (10), TruePeakLevel[1][hor2][vir2] indicates the true peak level value as loudness information of division point DV31 selected for CVP2 in Fig. 14. TruePeakLevel[2][hor3][vir3] indicates the true peak level value as loudness information of division point DV42 selected for CVP3 in Fig. 14.
[0251] The DRC processing unit 66 uses the sample peak level value EstSamplePeakLevel or the true peak level value EstTruePeakLevel calculated as above as loudness information to perform level correction of the output signal by DRC processing or the like.
[0252] <Description of output signal generation process> When loudness information is obtained by performing interpolation process using the normalized dependency of each CVP and the loudness information is used to correct the level of the output signal, the client 51 performs output signal generation process, for example, as shown in Figure 15.
[0253] The output signal generation process by the client 51 will be described below with reference to the flowchart of Fig. 15. Note that the processes of steps S191 to S193 are similar to the processes of steps S111 to S113 in Fig. 11, and therefore description thereof will be omitted.
[0254] In step S194, the loudness information processing unit 65 calculates the dependency ratio, that is, the degree of dependency, of each CVP based on the current listener position.
[0255] Specifically, the loudness information processing unit 65 calculates the distance from each CVP to the listener in the virtual space based on the supplied listener position information and the CVP position information included in the system configuration information from the bitstream decoder 62.
[0256] Then, the loudness information processing unit 65 calculates the distance ratio, that is, the normalized dependency for each CVP, by performing a calculation similar to that of the above-mentioned equation (8) based on the calculated distance.
[0257] In step S195, the loudness information processing unit 65 acquires listener orientation information supplied in response to an input operation or the like by the listener.
[0258] In step S196, the loudness information processing unit 65 selects one of the multiple CVPs set in the virtual space as the CVP to be processed, based on the system configuration information.
[0259] Once the CVP to be processed is selected, steps S197 to S199 are performed on the CVP to be processed. These steps are similar to steps S116 to S118 in FIG. 11, and therefore will not be described further.
[0260] As a result, for the CVP to be processed, loudness information loudnessInfo[i][sel_hidx][sel_vidx] determined by the combination of the horizontal division index sel_hidx and the vertical division index sel_vidx is read from the multi-angle loudness information. The loudness information read in this way is the loudness information associated with the division point located in the direction closest to the listener's orientation in the CVP to be processed.
[0261] In step S200, the loudness information processing unit 65 determines whether or not the processes in steps S197 to S199 have been performed for all CVPs.
[0262] If it is determined in step S200 that not all CVPs have been processed, the process returns to step S196 and the above-described process is repeated. In this case, a new CVP that has not yet been processed is selected as the processing target, and the above-described process is performed.
[0263] On the other hand, if it is determined in step S200 that the processing has been performed for all CVPs, the processing then proceeds to step S201. In this case, for all CVPs, loudness information for a direction closest to the listener direction indicated by the listener direction information is selected from the loudness information for each of the multiple directions (orientations) in the CVP.
[0264] In step S201, the loudness information processing unit 65 calculates loudness information according to the position and orientation of the listener based on the dependency ratio (degree of dependency) calculated in step S194 and the loudness information of each CVP read out in step S199. The loudness information processing unit 65 also supplies the calculated loudness information to the DRC processing unit 66.
[0265] For example, the loudness information processing unit 65 performs calculations similar to those of the above-mentioned equations (9) and (10) to calculate sample peak level values and true peak level values as loudness information by interpolation processing based on the degree of dependency (distance ratio).
[0266] After the loudness information is calculated, the processes of steps S202 to S205 are performed, and the output signal generation process ends. However, since these processes are similar to the processes of steps S119 to S122 in FIG. 11, their description will be omitted.
[0267] In this way, the client 51 selects appropriate loudness information for each CVP according to the listener's orientation, and calculates loudness information according to the listener's position and orientation by interpolation based on the selected loudness information.The client 51 then uses the calculated loudness information to correct the level of the output signal.
[0268] By doing this, not only can the occurrence of clipping distortion be suppressed, but level correction based on loudness information can be achieved in DRC processing, etc., and the playback volume of content can be prevented from becoming too quiet.
[0269] Fifth Embodiment Regarding the Present Technology, there are known two types of audio playback: 3DoF (Degree of Freedom) audio, in which the listener's position is fixed and the listener's orientation can be freely changed, and 6DoF audio, in which the listener's position and orientation can be freely changed. 6DoF audio is also called free viewpoint audio.
[0270] When performing loudness control for a single 3DoF audio, loudness values are measured based on the output to each speaker obtained by the rendering process. On the playback side, the difference between the measured loudness value and the target loudness value requested by the listener is calculated, and this gain difference is applied to the rendered output audio data.
[0271] On the other hand, in a free viewpoint audio system, the position and gain of objects at any viewpoint are determined by interpolation using 3DoF audio data produced from multiple viewpoints in three-dimensional space (CVP).However, there is no method available to handle the target loudness values required on the playback side for multiple 3DoF audio.
[0272] As a result, it is not possible to control the playback loudness in free viewpoint audio (6DoF audio).
[0273] Therefore, in this technology, loudness control can be performed on the playback side even in free viewpoint audio.
[0274] This technology has the following features, for example:
[0275] That is, the present technology has a feature of performing loudness correction only by controlling the gain of the object.
[0276] Furthermore, this technology has the advantage of being able to realize free viewpoint audio playback that matches the target loudness value on the playback side, even when there is variation in volume depending on the listener's position (viewpoint).
[0277] For example, when producing content, the loudness value for the rendering result for each viewpoint (CVP) is measured as a measured loudness value, and the obtained measured loudness value is stored in a bitstream as part of the configuration information and transmitted to the playback side. Note that the speaker output signal to be measured when calculating the measured loudness value in this embodiment is, for example, for all CVPs, when the listener is facing a TP (target point) described later.
[0278] On the playback side, the listener's desired loudness value is input as the target loudness value. The 6DoF metadecoder calculates the gain correction amount for the difference between the target loudness value and the measured loudness value for each CVP before interpolating the gain value of the object for a given listener position. The gain correction amount is then applied to the gain values included in the metadata of all objects for each CVP.
[0279] This makes it possible to play free viewpoint audio according to the target volume (target loudness value) on the playback side, even if there is a difference in volume between viewpoints (CVP).
[0280] Furthermore, this technology has the advantage of being able to realize free viewpoint audio playback that reflects the intentions of the content creator and the target loudness value.
[0281] For example, it is conceivable that a content creator may set the desired loudness value in advance as the production loudness value for each CVP, so if it is possible to accommodate such cases, it will be possible to better reflect the creator's intentions.
[0282] In this case, the volume relationship between CVPs set by the producer is maintained, and the difference between the CVP with the largest loudness value among them and the target loudness value is calculated, and that difference is applied to the gain value of the object for all CVPs.
[0283] When loudness control is performed on the playback side in free viewpoint audio, processing corresponding to each case of Case Index=0, 1, 2, 3 is performed on the playback side, for example, as shown in FIG.
[0284] FIG. 16 shows, for each case, the operating specifications, the presence or absence of group information and production loudness values in the configuration information (Config Info), and the settings on the playback side (loudness setting API (Application Programming Interface)).
[0285] For example, a loudness mode is set on the playback side. The loudness mode is an operation mode in which loudness correction is performed on the playback side based on a target loudness value set by a user (listener) or the like. For example, when a target loudness value is specified by a listener or the like on the playback side, the loudness mode is set.
[0286] The loudness modes include a measurement loudness mode (Case Index = 1), a group mode (Case Index = 2), and a production loudness mode (Case Index = 3).
[0287] When Case Index=0 (hereinafter also referred to as Case 0), the target loudness value is not specified, so the loudness mode is not set and no particular loudness correction is performed on the playback side.
[0288] On the other hand, when a target loudness value is specified and a loudness mode is selected, loudness correction is performed in one of the measured loudness mode, group mode, and production loudness mode.
[0289] In Case Index=1 (hereinafter also referred to as Case 1), i.e., in the measured loudness mode, it is not necessary to store group information or a production loudness value in the configuration information. In other words, in the measured loudness mode, group information or a production loudness value is not used.
[0290] In the measured loudness mode, even if there is variation in loudness (volume) at each CVP, which is the control viewpoint, the variation is made uniform, and loudness correction is performed so that the loudness during playback becomes the target loudness value set by the listener, etc. As will be described later, in the measured loudness mode, the gain information of the object is corrected based on the measured loudness value and the target loudness value.
[0291] In Case Index=2 (hereinafter also referred to as Case 2), i.e., in group mode, group information is stored in the configuration information, but it is not necessary to store the production loudness value in the configuration information. In other words, in group mode, group information is used.
[0292] The group information indicates the group (hereinafter referred to as the CVP group) to which each of the multiple CVPs (control viewpoints) arranged in the virtual space belongs. In group mode, there are cases where the number of groups is 1. In other words, there are cases where all CVPs belong to the same group.
[0293] In group mode, loudness correction is performed so that the loudness during playback becomes the target loudness value, based on the CVP with the largest measured loudness value in the same group, while maintaining the volume variation among each CVP belonging to the same group. As will be described later, in group mode, the gain information of the object is corrected based on the measured loudness value, group information, and target loudness value.
[0294] In Case Index = 3 (hereinafter also referred to as Case 3), i.e., in the production loudness mode, the composition information stores the production loudness value, but it is not necessary to store group information in the composition information. In other words, in the production loudness mode, the production loudness value specified for each CVP by the producer or the like is used.
[0295] In the production loudness mode, the production loudness value of each CVP (control viewpoint) is used to perform loudness correction so that the loudness during playback matches the target loudness value. That is, in the production loudness mode, the gain information of the object is corrected based on the measured loudness value, the production loudness value, and the target loudness value.
[0296] <About Loudness Modes> The measurement loudness mode, group mode, and production loudness mode will be specifically described below with reference to Figures 17 to 22. Note that in Figures 17 to 22, descriptions of corresponding parts will be omitted as appropriate.
[0297] (Measured loudness mode) In measured loudness mode (case 1), when the measured loudness values for each CVP are different, gain correction processing is performed on the playback side so that the loudness value during playback becomes the target loudness value regardless of the listener's position (listening position).
[0298] For example, as shown on the left side of FIG. 17, it is assumed that a target point (hereinafter also referred to as a TP) and a plurality of CVPs are defined in a virtual space.
[0299] The TP (Target Point) is a predetermined reference position, and as an example, the object metadata is generated assuming that all virtual listeners in each CVP are facing the TP. Here, the circle with the letter "TP" written on it represents the TP.
[0300] The metadata of the object in each CVP is not limited to that when the virtual listener in the CVP faces the TP, but can be that when the listener faces any direction.
[0301] In this example, five CVPs are placed, CVP A to CVP E. The letter written inside the circle indicates which CVP it is, for example, the circle with the letter "A" written inside it represents CVP A.
[0302] On the content production side, object metadata is defined for each CVP. For example, the metadata for each CVP includes information such as the position and gain of the object defined for that CVP. Also, even for the same object, the object's placement position in the virtual space and gain information may differ for each CVP.
[0303] The content creator uses the metadata to measure the loudness of the audio data of the content for each CVP as a measured loudness value. The audio data of the content is data for reproducing the sound of the content, including the sound of one or more objects.
[0304] For example, object metadata includes object position information and gain information for each CVP. Rendering is performed for each CVP using the object position information and gain information, and audio data with an arbitrary channel configuration, such as 5ch, 2ch, or 13ch, is generated. This audio data is used to play the sound of the content when the position of the CVP is the listener's position.
[0305] Furthermore, the loudness value of the audio data generated for each CVP is measured, and the measurement result is set as the measured loudness value for each CVP.
[0306] In this example, the measured loudness value La of CVP A, the measured loudness value Lb of CVP B, the measured loudness value Lc of CVP C, the measured loudness value Ld of CVP D, and the measured loudness value Le of CVP E are obtained by measurement.
[0307] On the other hand, on the playback side, a target loudness value Lt[LKFS] is set as the loudness value desired by the listener.
[0308] In the measured loudness mode, for each CVP, the loudness value of the audio data obtained by rendering processing on the playback side is set to the target loudness value Lt. In other words, the gain information of each CVP of each object is corrected (gain control) so that all the loudness values in each CVP become the target loudness value Lt, thereby realizing loudness correction.
[0309] To achieve such loudness correction, on the playback side, a loudness change value for correcting the gain information (gain value) of an object is calculated for each CVP, as shown on the right side of FIG.
[0310] In this example, a loudness change value Ga for CVP A, a loudness change value Gb for CVP B, a loudness change value Gc for CVP C, a loudness change value Gd for CVP D, and a loudness change value Ge for CVP E are determined.
[0311] For example, these loudness change values Ga to Ge can be obtained by the following equation (11).
[0312]
[0313] In equation (11), the difference between the target loudness value Lt and the measured loudness value, that is, the value obtained by subtracting the measured loudness value from the target loudness value Lt, is calculated as the loudness change value.
[0314] Also, on the playback side, a gain change rate for correcting the gain information of the object (gain correction) is calculated based on the loudness change value obtained for each CVP, and the gain information of each CVP of the object is corrected based on the obtained gain change rate.
[0315] For example, in the example of FIG. 17, the gain change rates GaRatio to GeRatio of each CVP of CVP A to CVP E are found by calculating the following equation (12) based on the loudness change value of each CVP.
[0316]
[0317] The gain change rate for each CVP obtained in this way is multiplied by the gain information for each CVP of the object to obtain the final gain information for each CVP of the object.
[0318] For example, in the example of Figure 17, the final object gain information ObgGain_a[i] to ObgGain_e[i] of each CVP A to CVP E is obtained by calculating the following equation (13) based on the gain change rate of each CVP.
[0319]
[0320] In equation (13), ObgGain_a[i] to ObgGain_e[i] indicate the gain information (gain value before correction) of the i-th object for CVP A to CVP E included in the metadata. Therefore, in equation (13), the value obtained by multiplying the gain information of the object by the gain change rate is the final gain information of the object, that is, the gain information after correction (hereinafter also referred to as correction gain information).
[0321] During rendering processing on the playback side, the correction gain information obtained for each CVP is used to calculate gain information for each object relative to the listener's position, and this information is applied to the audio data of each object.
[0322] (Group Mode) In group mode, CVPs are divided into groups, and the relative relationship of loudness values between CVPs included in the same group is maintained.
[0323] On the content production side, for example, five CVPs, CVP A to CVP E, are set as shown in FIG. 18, and these CVPs are grouped so that each CVP belongs to one of the groups (CVP groups).
[0324] In this example, CVP C and CVP D form one group 1, and CVP A, CVP B, and CVP E form one group 2.
[0325] For example, a creator may group CVPs for which a relative relationship is desired, in which the difference between the largest measured loudness value and the CVP's own measured loudness value is added to the loudness change value of the CVP for which the loudness value is to be intentionally lowered. In other words, the grouping is performed so that CVPs for which the creator wishes to maintain the volume balance (relative relationship of volume) are grouped together.
[0326] As will be described later, if the largest of the measured loudness values of each CVP belonging to the same group is set as the maximum measured loudness value, then when calculating the loudness change value, that maximum measured loudness value is set (used) as the measured loudness value of each CVP in the group. By doing so, on the playback side, the relative relationship of the measured loudness values of each CVP belonging to the same group, that is, the relative relationship of the volume (relationship of gain difference) is maintained within the same group.
[0327] For example, if a producer wants to maintain a balanced relationship in volume among CVP A, CVP B, and CVP E, the producer may group CVP A, CVP B, and CVP E into one CVP group, "Group 2," as shown in Fig. 18. In this case, if the largest of the measured loudness values of CVP A, CVP B, and CVP E is the measured loudness value Lb of CVP B, then when calculating the loudness change value, not only CVP B but also CVP A and CVP E are treated as having a measured loudness value of Lb.
[0328] For example, in this technology, the concept of a CVP group is provided, and information (group information) indicating the group to which each CVP belongs is stored in the bitstream, but even in group mode, the original measured loudness value of each CVP may be stored in the bitstream. This makes it possible to accommodate cases where the playback side wants to set the loudness values of all CVPs to the target loudness value, that is, to make the volume of all CVPs uniform in measured loudness mode.
[0329] In group mode, the playback side determines the maximum measured loudness value within each CVP group based on the group information value defined for each CVP, and sets the maximum measured loudness value as the measured loudness value for all CVPs belonging to the group.
[0330] It should be noted that the grouping of CVPs may be performed, for example, according to a designation operation by the creator, but may be performed by any other method.
[0331] As an example, clustering may be performed based on the placement positions of multiple CVPs in a virtual space, or grouping may be performed based on the distance between CVPs so that CVPs with short distances between them belong to the same group.
[0332] Furthermore, for example, the group to which a CVP belongs may be determined depending on the area in the virtual space in which the CVP is placed.
[0333] For example, if the virtual space is a space that includes a live venue and an area outside the venue, CVPs located within the venue may belong to the same group, while CVPs located outside the venue may belong to a different group than the group to which the CVPs within the venue belong. Also, even within the same venue, CVPs in the first floor seats may belong to the same group, while CVPs in the second floor seats may belong to a different group than the CVPs in the first floor seats.
[0334] The number of groups and the grouping may also be dynamically changed over time. Furthermore, a plurality of grouping results (group patterns) may be prepared in advance, such as group information for area A when the listener is in area A in the virtual space, and group information for area B when the listener is in area B.
[0335] In such a case, the group pattern may be switched so that one of a plurality of group patterns is selected depending on, for example, the location of the listener, the resources on the playback side, the remaining battery level on the playback side, the device type on the playback side, and the state of the network such as congestion.
[0336] The group pattern may be switched by the server that distributes the content, or by the content playback side (client). When group patterns are switched in this way, it is possible to prepare configuration information (described later) for each group pattern, for example.
[0337] In the group mode, all CVPs may belong to the same group, as shown on the left side of Fig. 19. In other words, all CVPs in the virtual space may belong to one group.
[0338] 19, five CVPs, CVP A to CVP E, are all in the same group. In this example, the relationship between the measured loudness values of all CVPs is maintained on the playback side, and the difference between the largest measured loudness value in the group and the target loudness value is applied to all CVPs in the same group.
[0339] Specifically, when correcting the gain information for each CVP of an object on the playback side, a loudness change value is first calculated.
[0340] In the group mode, the maximum value among the measured loudness values of each CVP belonging to the group is specified as the maximum measured loudness value, and the difference between the maximum measured loudness value and the target loudness value is set as the loudness change value. In particular, in the group mode, the loudness change values of all CVPs belonging to the group are the same value.
[0341] For example, in the example of FIG. 19, CVPs A to E belong to the same group, and the maximum measured loudness value of these CVPs is the measured loudness value Ld of CVP D.
[0342] Therefore, the maximum measured loudness value Lmax = Ld, and as shown on the right side of the figure, the loudness change values Ga to Ge of CVP A to CVP E are calculated based on the maximum measured loudness value Lmax and the target loudness value Lt using the following equation (14).
[0343]
[0344] In equation (14), the difference between the target loudness value Lt and the maximum measured loudness value Lmax, i.e., the value obtained by subtracting the maximum measured loudness value Lmax from the target loudness value Lt, is calculated as the loudness change values Ga to Ge. In particular, the loudness change values Ga to Ge of CVPs A to E belonging to the same group are the same value.
[0345] Also, when the number of groups is two as shown in FIG. 18, a loudness change value is calculated for each group as shown in FIG. 20, for example.
[0346] On the left side of FIG. 20, the same groupings as in FIG. 18 are used, with group 1 consisting of CVP C and CVP D, and group 2 consisting of CVP A, CVP B, and CVP E.
[0347] In this case, the larger (largest) of the measured loudness value Lc of CVP C and the measured loudness value Ld of CVP D is set as the maximum measured loudness value Lmax_g1 of group 1.
[0348] Similarly, the largest one (maximum value) among the measured loudness value La of CVP A, the measured loudness value Lb of CVP B, and the measured loudness value Le of CVP E is set as the maximum measured loudness value Lmax_g2 of group 2.
[0349] The difference between the target loudness value Lt and the maximum measured loudness value Lmax_g1 is set as the loudness change value for all CVPs in group 1, and the difference between the target loudness value Lt and the maximum measured loudness value Lmax_g2 is set as the loudness change value for all CVPs in group 2.
[0350] That is, as shown on the right side of FIG. 20, the loudness change values Gc=Gd=Lt-Lmax_g1, and the loudness change values Ga=Gb=Ge=Lt-Lmax_g2.
[0351] For example, in the example of Figure 20, similar to the case of the above-mentioned equation (12), the gain change rates GaRatio to GeRatio of each CVP of CVP A to CVP E can be found by calculating the following equation (15) based on the loudness change value of each CVP.
[0352]
[0353] The gain change rate for each CVP obtained in this way is multiplied by the gain information for each CVP of the object to obtain the final gain information for each CVP of the object.
[0354] In the example of Figure 20, similar to the above-mentioned equation (13), by calculating the following equation (16) based on the gain change rate of each CVP, corrected gain information ObgGain_a[i] to ObgGain_e[i], which is the gain information of the final object of each CVP of CVP A to CVP E, is obtained.
[0355]
[0356] In equation (16), ObgGain_a[i] to ObgGain_e[i] indicate the gain information (gain value before correction) of the i-th object for CVP A to CVP E included in the metadata. In equation (16), the value obtained by multiplying the gain information of the object by the gain change rate is set as the final gain information (corrected gain information) of the object.
[0357] During rendering processing on the playback side, the correction gain information obtained for each CVP is used to calculate gain information for each object relative to the listener's position, and this information is applied to the audio data of each object.
[0358] (Production Loudness Mode) In the production loudness mode, the production loudness value of each CVP intended by the producer when producing the content is set.
[0359] For example, when CVPs A to E are arranged in a virtual space as shown on the left side of FIG. 21, a production loudness value is determined for each of the CVPs, and a measured loudness value is measured for each CVP.
[0360] In this example, the produced loudness values Lca to Lce of CVP A to CVP E are set, and the measured loudness values La to Le of CVP A to CVP E are obtained by measurement.
[0361] By specifying (setting) the production loudness value, producers can intentionally set the volume balance between CVPs.
[0362] In the production loudness mode, the loudness change value is calculated in two stages.
[0363] That is, in the first stage of processing, an intermediate loudness modification value is calculated for each CVP as shown in the center of the figure.
[0364] Specifically, the intermediate loudness change value is determined by subtracting the measured loudness value from the production loudness value, as shown in the following equation (17): In equation (17), Ga to Ge represent the intermediate loudness change values of CVP A to CVP E.
[0365]
[0366] For example, if the intermediate loudness modification value Ga is used as the loudness modification value, loudness correction is performed for CVP A so that the loudness during playback becomes the production loudness value Lca.
[0367] Next, in the second stage of processing, as shown on the right side of the figure, a common correction amount OvaG, which is a correction amount common to all CVPs, is calculated, and based on this common correction amount OvaG, a final loudness change value, which is the final loudness change value for each CVP, is calculated.
[0368] Specifically, the maximum value among the production loudness values of all CVPs, that is, the maximum value among the production loudness values Lca to Lce, is set as the maximum production loudness value Lcx_max.
[0369] Furthermore, as shown in the following equation (18), the difference between the target loudness value Lt and the maximum produced loudness value Lcx_max, that is, the value obtained by subtracting the maximum produced loudness value Lcx_max from the target loudness value Lt, is set as the common correction amount OvaG.
[0370]
[0371] The common correction amount OvaG is the correction amount for the intermediate loudness change value of each CVP so that the loudness in the CVP that results in the maximum production loudness value Lcx_max becomes the target loudness value Lt and the volume balance between each CVP, i.e., the relative volume (loudness) relationship, is maintained.
[0372] Finally, a final loudness change value (final loudness change value) is calculated for each CVP based on the common correction amount OvaG and the intermediate loudness change value.
[0373] That is, the final loudness change values fGa to fGe of the CVPs A to E are calculated by the following equation (19).
[0374]
[0375] In equation (19), the common correction amount OvaG is added to the intermediate loudness change value for each CVP, and the result of this addition is set as the final loudness change value.
[0376] FIG. 22 shows specific examples of the final loudness change values fGa to fGe of each CVP shown in FIG.
[0377] In this example, the production loudness values of CVP A to CVP E are all set to "-3", so the maximum production loudness value Lcx_max = -3. Also, -11 is specified as the target loudness value Lt, so the common correction amount OvaG = -8.
[0378] Thus, for example, the final loudness modification value fGe for CVP E with a measured loudness value Le=-19.75 will be 8.75.
[0379] In the example of FIG. 21, once the final loudness change value of each CVP is obtained, the gain change rates GaRatio to GeRatio of each CVP of CVP A to CVP E are calculated using the final loudness change value according to the following equation (20).
[0380]
[0381] Furthermore, the gain change rate for each CVP is multiplied by the gain information for each CVP of the object to obtain the final gain information for each CVP of the object.
[0382] In the example shown in FIG. 21, similar to the above-mentioned formula (13), the corrected gain information ObgGain_a[i] to ObgGain_e[i], which is the gain information of the final object of each CVP of CVP A to CVP E, is obtained by calculating the following formula (21) based on the gain change rate of each CVP.
[0383]
[0384] In the production loudness mode, if the obtained correction gain information is used, the loudness in the CVP that results in the maximum production loudness value Lcx_max becomes the target loudness value Lt. Furthermore, the relative magnitude relationship of the loudness between each CVP is the same as the relative magnitude relationship of the production loudness values of each CVP. In the production loudness mode, a target loudness value is set, but even if a target loudness value is not set, the intention of the producer may be reflected. In that case, processing as Lt = Lcx_max makes it possible to deal with cases where a target loudness value is not set.
[0385] With the present technology as described above, loudness correction can be performed only by controlling the gain of the object. That is, by controlling the gain of the object, it is possible to appropriately control loudness at any listener position in a virtual space such as a three-dimensional space.
[0386] For example, in the measured loudness mode, even if the volume balance between CVPs was not taken into consideration when the content was created, the loudness at each viewpoint (listening position) on the playback side can be adjusted to match the target loudness value.
[0387] Also, for example, in group mode, for each group set during content creation, the loudness at the CVP with the maximum measured loudness value can be adjusted to the target loudness value while maintaining the volume balance between CVPs within the group.
[0388] Furthermore, for example, in production loudness mode, the loudness at the CVP where the production loudness value is maximum can be adjusted to the target loudness value while maintaining the volume balance between CVPs desired by the content creator.
[0389] <Processing Flow on the Production Side and Playback Side> The general processing flow on the content production side and playback side will be described.
[0390] When producing content, for example, as shown in the upper part of Figure 23, the production side's device realizes a 6DoF decoder 101, a renderer 102, a loudness measurement tool 103, and a configuration information generation unit 104 as functional processing blocks, and configuration information is generated.
[0391] Specifically, the 6DoF decoder 101 decodes the metadata for each CVP of each object for one or more CVPs in the virtual space.
[0392] The renderer 102 performs rendering processing such as VBAP based on the metadata for each CVP of each object and the audio data for playing the sound of each object, thereby generating audio data with a desired channel configuration, such as 2ch or 21ch. The renderer 102 performs rendering processing for each CVP. In the rendering processing, the position of the CVP is used as the position of the listener, and an arbitrary direction is used as the direction of the listener.
[0393] The loudness measurement tool 103 measures loudness based on audio data of a desired channel configuration obtained for each CVP, and outputs the measurement result as a measured loudness value for each CVP.
[0394] The configuration information generating unit 104 generates information related to the loudness of the CVPs, including the measured loudness value for each CVP, as multi-loudness information, and also generates and outputs configuration information including the multi-loudness information and other information related to the CVPs.
[0395] The playback side is supplied with the configuration information thus obtained, the metadata for each CVP of each object, and the audio data of each object.
[0396] In the playback device, as shown at the bottom of the figure, a 6DoF decoder 121, a rendering module 122, and an audio output module 123 are realized as functional processing blocks, and audio data of the content is output.
[0397] Specifically, the 6DoF decoder 121 performs decoding and interpolation processing on the configuration information and the metadata for each CVP of the object, and generates metadata for each object based on the position of the listener (hereinafter also referred to as listener-based metadata).
[0398] The generation of listener-reference metadata also uses listener position information indicating the position of the listener in the virtual space, listener direction information indicating the direction of the listener's face in the virtual space, i.e., the direction of their line of sight, and target loudness values specified by the listener, etc.
[0399] The rendering module 122 performs rendering processing such as VBAP based on the listener-reference metadata of each object and the audio data of each object, thereby generating audio data with a desired channel configuration such as 2ch or 21ch.
[0400] For example, the rendering module 122 performs rendering processing similar to that performed by the renderer 102. The audio data generated by the rendering module 122 is audio data (hereinafter also referred to as output audio data) for playing back the sounds of the content including the sounds of each object.
[0401] The audio output module 123 outputs the output audio data generated by the rendering module 122 to an audio output unit such as a speaker or headphones.
[0402] The configuration information generated on the content production side, more specifically, the multi-loudness information included in the configuration information, is, for example, the information shown in Fig. 24. That is, the multi-loudness information is described in the bitstream in the format shown in Fig. 24.
[0403] In this example, the multi-loudness information includes group mode flag information "LoudCvpGroupMode" that indicates whether or not the group mode is being used, that is, whether or not the group mode is being used.
[0404] For example, a value of "0" in the group mode flag information indicates that group mode is not used (not group mode), and a value of "1" in the group mode flag information indicates that group mode is used and a CVP group is set for each CVP.
[0405] A value of "2" in the group mode flag information indicates that the group mode is used and all CVPs are in the same CVP group.
[0406] When the value of the group mode flag information is "1", the multi-loudness information stores the same number of group indices as the number of CVPs, i.e., for each CVP, a group index "LoudCvpGroup[i]" indicating the CVP group to which the i-th CVP belongs. For example, the value of the group index LoudCvpGroup[i] ranges from 0 to 15. This group index corresponds to the group information described with reference to FIG. 16.
[0407] In addition, the multi-loudness information stores loudness information "loudnessInfoMp[i]" relating to the i-th CVP for the number of CVPs.
[0408] This loudness information "loudnessInfoMp[i]" includes the measured loudness value of the i-th CVP, measurementCount, and so on.
[0409] For example, when production loudness values, etc. are prepared for each CVP for multiple playback environments, such as for each channel configuration, measurementCount is count information indicating the number of playback environments for channel configurations, etc., i.e., the number of prepared production loudness values, etc. Note that the number of measured loudness values for each CVP may be the same as the number in measurementCount, i.e., for each playback environment.
[0410] When the value of the group mode flag information is "0", the multi-loudness information stores production loudness value present flag information "CvpLoudValuePresentFlag".
[0411] The production loudness value presence flag information "CvpLoudValuePresentFlag" is flag information indicating whether or not a production loudness value is present, that is, whether or not a production loudness value is included in the multi-loudness information (composition information).
[0412] For example, a value of "0" in the production loudness value presence flag information indicates that a production loudness value does not exist (is not set), and a value of "1" in the production loudness value presence flag information indicates that a production loudness value exists (is set).
[0413] Therefore, it can be said that the value "0" of the production loudness value presence flag information indicates the measured loudness mode, and the value "1" of the production loudness value presence flag information indicates the production loudness mode.
[0414] When the value of the production loudness value existence flag information is "1", the multi-loudness information stores the production loudness values "CvpLoudValue[i][j]" for each CVP, the number of which is equal to the number of measurementCount, where i is the index of the CVP and j is the index of the playback environment corresponding to the measurementCount.
[0415] The group mode flag information and production loudness value presence flag information stored in the configuration information (multi-loudness information) can be said to be information for identifying whether the loudness mode is the measurement loudness mode, the group mode, or the production loudness mode.
[0416] Basically, the configuration information can be transmitted to the playback side, i.e., the client that plays the content, at a predetermined timing, such as before or when playback of the content starts. In other words, once the client obtains the configuration information, it can use that information to generate output audio data for each frame (time).
[0417] However, in some cases, the configuration information may change during playback of the content. For example, configuration information may be prepared for each area in the virtual space, each scene in the content, each client resource, each client's remaining battery level, each client device type, or each network status. In such cases, the configuration information used to generate the output audio data may be switched at any time during playback of the content.
[0418] An example in which configuration information is prepared for each area in the virtual space, i.e., for each viewing area where a listener is located, will be described with reference to Figures 25 and 26. Note that in Figure 25, parts corresponding to those in Figure 23 are given the same reference numerals, and their description will be omitted where appropriate.
[0419] For example, as shown on the left side of Figure 25, the server that distributes the content prepares configuration information for each of multiple viewing areas in the virtual space, such as configuration information for viewing area A to configuration information for viewing area C.
[0420] In such a case, the server appropriately acquires listener position information indicating the listener's current position in the virtual space from the client and selects configuration information. For example, if the listener's position indicated by the listener position information is a position within viewing area A, i.e., if the listener is located within viewing area A, the server selects configuration information for viewing area A.
[0421] Then, the configuration information selected according to the listener position, the metadata for each CVP of each object, and the audio data of each object are appropriately encoded and transmitted to the client. Here, it is assumed that the configuration information for viewing area A is transmitted to the client.
[0422] In the client, the 6DoF decoder 121 generates listener-reference metadata based on the configuration information acquired from the server and the metadata for each CVP of each object, in the same manner as in Fig. 23. In this example, the configuration information used is the configuration information for viewing area A.
[0423] Then, the rendering module 122 performs rendering processing based on the listener-reference metadata and audio data of each object, and generates output audio data of the content.
[0424] In this example, as shown in FIG. 26, for example, each time the viewing area where the listener is located is switched, that is, each time the listener moves to another viewing area, the configuration information transmitted to the client is switched.
[0425] In FIG. 26, the horizontal direction indicates time, that is, frames (time frames) of output audio.
[0426] In this example, for example, at a specific frame N, the listener moves to a viewing area different from the viewing area where he or she was previously located, so the configuration information of the viewing area to which the listener has moved, the metadata for each CVP of each object, and the audio data of each object are transmitted from the server to the client.
[0427] From frame (N+1) to frame (N+3), the listener did not move to another viewing area but remained in the same viewing area, so configuration information was not transmitted, and metadata for each CVP of each object and audio data for each object were transmitted from the server to the client.
[0428] Thereafter, the listener moves to another viewing area, that is, the viewing area is switched, so the configuration information is transmitted in frame (N+4).
[0429] That is, in frame (N+4), configuration information of the viewing area to which the listener has moved, metadata for each CVP of each object, and audio data of each object are transmitted from the server to the client.
[0430] In this way, if multiple types of configuration information are prepared for each viewing area, the configuration information to be transmitted can be switched as needed, thereby enabling content playback that better reflects the intentions of the content creator.
[0431] <Examples of display screens on the production side and playback side> Examples of display screens (UI (User Interface)) are shown in Figures 27 to 30. Note that in Figures 27 to 30, corresponding parts are given the same reference numerals, and their explanation will be omitted as appropriate.
[0432] 27 to 29 show examples of display screens (UI) displayed by a production tool for producing content, which are displayed on the content production side.
[0433] For example, on the display screen shown in FIG. 27, an image of a virtual space in which objects are arranged is displayed, and TPs and CVPs are arranged in the virtual space.
[0434] Here, a circle with the letter "TP" written on it represents a TP, and a circle with a letter such as "A" written on it represents a CVP. For example, a circle with the letter "A" written on it represents one CVP, CVP A. Also, a pentagonal mark LPT11 indicates the current position of the listener in the virtual space.
[0435] The content creator determines the placement positions of the TPs and CVPs by appropriately changing the position of the TPs and the positions of each CVP, adding a new CVP, deleting unnecessary CVPs, etc. Once the placement positions of the CVPs are determined, the loudness value is measured at each CVP as appropriate in response to the creator's operation, and the measurement results, i.e., the measured loudness value, are displayed in the vicinity of each CVP.
[0436] In this example, a display area CLR11 is displayed near CVP B, which is represented by a circle with the letter "B" inscribed therein. The display area CLR11 for CVP B displays the measured loudness value of CVP B, "-10.75."
[0437] In this way, producers can check the measured loudness values for each CVP displayed, and can also adjust the position of the CVP, etc., as appropriate, depending on the displayed measured loudness values.
[0438] Furthermore, in the production tool, as shown in FIG. 28, for example, it is possible to specify the CVP group of each CVP when producing content.
[0439] In the example on the left side of Fig. 28, a virtual space is displayed on the display screen, and in the virtual space, TPs, CVPs, the display areas of each CVP, and a mark LPT11 indicating the listener's position are displayed, as in Fig. 27. In this example, the display area of each CVP displays not only the measured loudness value of the CVP, but also information indicating the group to which the CVP belongs (CVP group).
[0440] For example, the display area CLR11 for CVP B displays the measured loudness value of CVP B, "-10.75," as well as "GP2," which is information indicating the group to which CVP B belongs (group name).
[0441] The creator can specify a group for each CVP by manipulating the display area. For example, as shown on the right side of the figure, the creator can display the group list GPL11, which is a user interface for selecting (specifying) a group for the CVP, by manipulating the part of the display area CLR12 of the desired CVP where the group name is displayed.
[0442] The group list GPL11 lists the CVP groups to which a CVP can belong, along with check boxes. Creators can specify (select) the group to which a CVP belongs by manipulating the check boxes to display a check mark in the check boxes. In this example, a check mark appears in the check box for the group name "GP2," indicating that the group name "GP2" has been specified.
[0443] In this way, in the production tool, a CVP group can be easily specified simply by performing operations on the display area of each CVP. Note that the method for specifying a CVP group is not limited to the example in Figure 28, and any other method, such as a drag operation, may be used.
[0444] In the production tool, as shown in FIG. 29, for example, it is possible to specify the production loudness value of each CVP when producing content.
[0445] The display screen shown in Fig. 29 displays a virtual space, in which TPs, CVPs, the display areas of each CVP, and a mark LPT11 indicating the listener's position are displayed, as in Fig. 27. In this example, the display area of each CVP displays not only the measured loudness value of the CVP, but also the produced loudness value of the CVP.
[0446] For example, the display area CLR11 of CVP B displays the measured loudness value of CVP B, "-10.75," as well as the produced loudness value of CVP B, "-6.75," specified (input) by the producer. The producer can input a produced loudness value for each CVP by operating the display area of the CVP as appropriate. Note that the method for inputting the measured loudness value is not limited to the example in FIG. 29 , and any other method may be used.
[0447] FIG. 30 shows an example of a display screen displayed by an application program that performs processing related to content playback on the playback side, that is, on the client side.
[0448] In this example, an image of a three-dimensional virtual space is displayed on the display screen, and in the virtual space, a mark LPT21 indicating the positions of TP, CVP, objects and listeners, and a display area RP11 relating to content playback are displayed.
[0449] Here, a sphere TPM11 in virtual space represents a TP, and images of sound source objects are displayed around the TP. Also, spheres with letters such as "A" represent CVPs. For example, a sphere with the letter "A" represents one CVP, CVP A.
[0450] By viewing the virtual space on the display screen, the listener can grasp the position of his / her viewpoint in the virtual space, the placement of CVPs, the placement of objects, etc. For example, the position of the listener, the number and positions of objects, and the number and positions of CVPs may change over time.
[0451] Display region RP11 is provided with a button group BT11 for controlling playback of content, a check box BX11, and an input field IPB11.
[0452] Button group BT11 includes a play button for instructing the start of content playback, a pause button for instructing the pause of content playback, and a play stop button for instructing the stop of content playback. A listener can start or stop content playback by operating the buttons in button group BT11.
[0453] The check box BX11 is operated when setting the loudness mode. For example, a listener can operate the check box BX11 during content playback to display a check mark in the check box BX11, thereby setting the loudness mode.
[0454] During loudness mode playback, the content is played back in either measured loudness mode, group mode, or produced loudness mode according to the configuration information.
[0455] The input field IPB11 is an area for inputting a target loudness value used in the loudness mode.
[0456] For example, when a check mark is displayed in the check box BX11 and the loudness mode is selected, the input field IPB11 becomes active, and it becomes possible to input a target loudness value into the input field IPB11.
[0457] The listener can input any value as the target loudness value into the input field IPB11 by operating the input field IPB11. In this example, "-6.75" is input as the target loudness value into the input field IPB11.
[0458] Alternatively, the input field IPB11 may be set to be always active, and when a target loudness value is input into the input field IPB11, a check mark may be displayed in the check box BX11, and the loudness mode may be selected. Alternatively, the check box BX11 may not be provided in the display area RP11, and when a target loudness value is input into the input field IPB11, the loudness mode may be selected.
[0459] <Configuration Example of Information Processing Device> Next, the configuration and operation of each device on the content production side, distribution side, and playback side will be described.
[0460] FIG. 31 is a diagram showing an example of the configuration of an embodiment of an information processing device that is a device on the content production side.
[0461] An information processing device 161 shown in FIG. 31 is, for example, a personal computer, and generates content configuration information and the like in response to operations by the creator.
[0462] The information processing device 161 has an input unit 171 , a display unit 172 , a communication unit 173 , a control unit 174 , a recording unit 175 , and an audio output unit 176 .
[0463] The input unit 171 is made up of, for example, a mouse and a keyboard, and supplies signals according to operations by the content creator to the control unit 174. The display unit 172 is made up of a display, and displays the display screen of the production tool, etc., under the control of the control unit 174.
[0464] The communication unit 173 communicates with an external device under the control of the control unit 174. For example, the communication unit 173 transmits configuration information, object metadata, and the like supplied from the control unit 174 to a server, which is an external device.
[0465] The control unit 174 controls the overall operation of the information processing device 161. The control unit 174 implements a rendering processing unit 181 and a loudness measuring unit 182 by executing the program of the production tool.
[0466] The rendering processing unit 181 performs rendering processing based on metadata and audio data of an object. The loudness measurement unit 182 performs loudness measurement based on the audio data of the content generated by the rendering processing.
[0467] The rendering processing unit 181 and the loudness measurement unit 182 correspond to, for example, the renderer 102 and the loudness measurement tool 103 shown in Fig. 23. The control unit 174 also functions as the 6DoF decoder 101 and the configuration information generation unit 104 shown in Fig. 23.
[0468] The recording unit 175 is made up of a non-volatile memory or the like, and records various data such as production tool programs, and supplies the recorded data to the control unit 174 as appropriate. The audio output unit 176 is made up of a speaker or the like, and outputs sound based on audio data supplied from the control unit 174. Note that the audio output unit 176 may be provided outside the information processing device 161. In such a case, headphones, earphones, or the like may be used as the audio output unit 176, for example.
[0469] <Description of Configuration Information Generation Process> For example, when a production tool is started, the control unit 174 controls the display unit 172 to cause the display unit 172 to display the display screen of the production tool.
[0470] With the production tool display screen displayed, the producer operates the input unit 171 to specify the position of the CVP and metadata for each CVP of each object, and the control unit 174 determines the position of the CVP etc. in accordance with the signal supplied from the input unit 171, or generates metadata for each CVP of the object.
[0471] Furthermore, when the producer performs an operation related to loudness while the display screen of the production tool is displayed, the information processing device 161 performs the configuration information generation process shown in Fig. 32 to generate configuration information. Hereinafter, the configuration information generation process by the information processing device 161 will be described with reference to the flowchart of Fig. 32.
[0472] In step S301, the control unit 174 measures the loudness value of each CVP.
[0473] For example, when metadata for each CVP of each object is encoded and stored, the control unit 174 performs a decoding process on the encoded metadata.
[0474] Furthermore, the rendering processing unit 181 of the control unit 174 performs rendering processing for each CVP. That is, the rendering processing unit 181 generates audio data of the content when the position of the CVP is the listening position by performing rendering processing such as VBAP based on CVP position information indicating the position of the CVP in the virtual space, metadata about the CVP of each object, and object data of each object, with the position of the CVP as the listener position.
[0475] The loudness measurement unit 182 of the control unit 174 calculates (measures) the loudness value of the audio data of the content in the CVP based on the audio data of the content obtained for each CVP, and sets the calculation result as the measured loudness value for the CVP.
[0476] Furthermore, the control unit 174 causes the measurement results of the loudness value for each CVP to be displayed on the display unit 172. As a result, the display unit 172 displays, for example, a display screen shown in FIG.
[0477] In step S302, the control unit 174 stores the measured loudness values of each CVP obtained in step S301 in the configuration information.
[0478] For example, when configuration information including the multi-loudness information shown in Fig. 24 is generated, the control unit 174 stores the measured loudness value for each CVP obtained in step S301 in the loudness information loudnessInfoMp[i] of the configuration information being held and in the process of being generated. In addition, the control unit 174 also stores information such as measurementCount in the loudness information loudnessInfoMp[i] as necessary.
[0479] In step S303, the control unit 174 determines whether or not the mode is the group mode, based on a signal corresponding to the producer's operation supplied from the input unit 171. For example, if the producer selects the group mode in the loudness mode settings, it is determined in step S303 that the mode is the group mode.
[0480] If it is determined in step S303 that the mode is the group mode, then in step S304 the control unit 174 sets the group index values of all CVPs to 0. In other words, the CVP group is set as the default group.
[0481] For example, when configuration information including the multi-loudness information shown in Figure 24 is generated, the control unit 174 sets the value of the group index LoudCvpGroup[i] of each CVP in the configuration information that it is holding and is currently generating to 0.
[0482] In this case, the control unit 174 controls the display unit 172 to display, for example, the display screen shown in Fig. 28 on the display unit 172. The creator operates the input unit 171 as needed to perform operations on the display area of any CVP to display a group list, and selects the CVP group by specifying a desired group (group name) from the group list. In other words, the creator performs an operation to change the CVP group from the default group to a group specified by the creator as needed.
[0483] In step S305, the control unit 174 determines whether or not there is a CVP whose group index value has been changed by the creator.
[0484] For example, when a producer performs an operation to change the group of a CVP, i.e., an operation to change the value of the group index indicating the group of the CVP, and a signal corresponding to that operation is supplied from the input unit 171, the control unit 174 determines that there is a CVP whose group index value has been changed.
[0485] If it is determined in step S305 that there is a CVP whose group index value has been changed, then in step S306 the control unit 174 changes the value of the group index.
[0486] Specifically, for example, in response to a signal supplied from the input unit 171, the control unit 174 changes (updates) the value of the group index LoudCvpGroup[i] of the CVP that is instructed to be changed in the configuration information that it is holding and is in the process of generating, to the value specified by the producer.
[0487] After the process of step S306 is performed, the process returns to step S305, and the above-described process is repeated.
[0488] Also, if it is determined in step S305 that there is no CVP whose group index value has been changed, that is, if the selection of the group for each CVP has been completed, the process proceeds to step S311.
[0489] In this case, the control unit 174 stores information indicating the group mode in the configuration information that is being held and is currently being generated. For example, when generating configuration information including the multi-loudness information shown in Fig. 24, the control unit 174 sets the value of group mode flag information LoudCvpGroupMode in the configuration information to "1" or "2" depending on the value of the group index of each CVP.
[0490] If it is determined in step S303 that the mode is not the group mode, then in step S307 the control unit 174 determines whether or not the mode is the production loudness mode, based on a signal corresponding to the producer's operation supplied from the input unit 171. For example, if the producer selects the production loudness mode in the loudness mode settings, it is determined in step S307 that the mode is the production loudness mode.
[0491] If it is determined in step S307 that the production loudness mode is not selected, that is, if the measured loudness mode has been selected by the producer (if it is determined that the measured loudness mode is selected), then the process proceeds to step S311.
[0492] In this case, the control unit 174 stores information indicating that the mode is not group mode in the configuration information that is being held and is currently being generated. For example, when generating configuration information including the multi-loudness information shown in Fig. 24, the control unit 174 sets the value of group mode flag information LoudCvpGroupMode in the configuration information to "0." In addition, the control unit 174 sets the value of production loudness value presence flag information CvpLoudValuePresentFlag in the configuration information to "0."
[0493] On the other hand, if it is determined in step S307 that the production loudness mode is selected, the control unit 174 sets the production loudness values of all CVPs to 0 in step S308.
[0494] For example, when configuration information including the multi-loudness information shown in Fig. 24 is generated, the control unit 174 sets the value of the production loudness value presence flag information CvpLoudValuePresentFlag in the configuration information being held and in the process of being generated to "1." Furthermore, the control unit 174 sets the value of the production loudness value CvpLoudValue[i][j] of each CVP in the configuration information to 0.
[0495] In this case, the control unit 174 controls the display unit 172 to display, for example, the display screen shown in Fig. 29 on the display unit 172. The producer appropriately operates the input unit 171 to operate the display area of any CVP, thereby inputting the production loudness value of that CVP. In other words, the producer appropriately operates to change the production loudness value from the default value "0" to a value specified by the producer.
[0496] In step S309, the control unit 174 determines whether or not there is a CVP whose production loudness value has been changed by the producer.
[0497] For example, when a producer performs an operation to change the production loudness value of a CVP and a signal corresponding to that operation is supplied from the input unit 171, the control unit 174 determines that there is a CVP whose production loudness value has been changed.
[0498] If it is determined in step S309 that there is a CVP whose production loudness value has been changed, the control unit 174 changes the production loudness value in step S310.
[0499] Specifically, for example, the control unit 174 changes (updates) the production loudness value CvpLoudValue[i][j] of the CVP that is instructed to be changed in the configuration information being held and in the middle of generation to a value specified by the producer in response to a signal supplied from the input unit 171. Note that, as described above, a plurality of production loudness values, etc. stored in the configuration information (multi-loudness information) may be prepared for each playback environment, etc.
[0500] After the process of step S310 is performed, the process returns to step S309, and the above-described process is repeated.
[0501] Also, if it is determined in step S309 that there is no CVP whose production loudness value has been changed, that is, if the setting of the production loudness value for each CVP has been completed, the process proceeds to step S311.
[0502] In this case, the control unit 174 stores information indicating that a production loudness value is stored in the configuration information that is being held and is in the process of being generated. For example, when generating configuration information including the multi-loudness information shown in Fig. 24, the control unit 174 sets the value of the production loudness value presence flag information CvpLoudValuePresentFlag in the configuration information to "1".
[0503] If it is determined in step S305 that there is no CVP whose group index value has been changed, if it is determined in step S307 that the mode is measurement loudness mode, or if it is determined in step S309 that there is no CVP whose production loudness value has been changed, processing of step S311 is performed.
[0504] In step S311, the control unit 174 outputs the configuration information that it holds.
[0505] For example, the control unit 174 stores necessary information as appropriate in the configuration information including the multi-loudness information obtained by the processing up to this point to create the final configuration information, and outputs this final configuration information to the recording unit 175 for recording, thereby completing the configuration information generation processing.
[0506] In this case, the control unit 174 appropriately supplies metadata for each CVP of each object, audio data of each object, and the like to the recording unit 175 to be recorded.
[0507] The final configuration information also includes, for example, multi-loudness information, object number information indicating the number of objects that make up the content, CVP number information indicating the number of CVPs prepared in advance, and CVP information regarding the CVPs.
[0508] For example, the CVP information includes a CVP index, CVP position information, and CVP orientation information.
[0509] The CVP index is ID information that uniquely identifies the CVP. The CVP position information is position information that indicates the absolute position of the CVP in the virtual space, and the CVP orientation information is information that indicates the orientation of the face of the virtual listener in the CVP in the virtual space. For example, the CVP orientation information can be information that indicates the direction from the CVP to the TP.
[0510] As described above, the configuration information may be generated for each viewing area, each content scene, each client resource, each client remaining battery level, each client device type, each network state, etc. In other words, the control unit 174 may generate a plurality of different pieces of configuration information.
[0511] Furthermore, for example, the control unit 174 reads the content configuration information, the metadata for each CVP of each object, and the audio data of each object from the recording unit 175 at any timing, and supplies these to the communication unit 173. The communication unit 173 transmits the content configuration information, the metadata for each CVP of each object, and the audio data of each object supplied from the control unit 174 to the server as content data.
[0512] As described above, the information processing device 161 selects (sets) a loudness mode in response to an operation by the producer, and generates configuration information including a measured loudness value and the like in response to the selection result and the like.
[0513] By doing this, the content playback side can control loudness only by controlling the gain of the object using the configuration information, thereby realizing free viewpoint audio playback that reflects the intentions of the producer and the target loudness value specified by the listener.
[0514] <Configuration Example of Server> FIG. 33 is a diagram showing a configuration example of an embodiment of a server to which the present technology is applied.
[0515] The server 211 shown in FIG. 33 is an information processing device such as a computer, and functions as an encoder that receives content data from the information processing device 161 and distributes the content data to clients.
[0516] The server 211 includes an acquisition unit 221 , a bitstream encoder 222 , and a communication unit 223 .
[0517] The acquisition unit 221 receives (acquires) content data transmitted from the information processing device 161 , that is, configuration information, metadata for each CVP of each object, and audio data of each object, and supplies this to the bitstream encoder 222 .
[0518] For example, the metadata for each CVP of an object includes object position information that indicates the position of the object in the virtual space, and gain information for the audio data of the object, which are defined for the CVP.
[0519] The object position information may be coordinate information expressed in polar coordinates that indicates the relative position of the object as seen from the CVP, or may be coordinate information that is determined for each CVP and expressed in absolute coordinates (orthogonal coordinates) that indicates the absolute position of the object in virtual space. The gain information is information on a gain value used for gain correction (gain adjustment) of the audio data of the object.
[0520] Additionally, the metadata for each CVP of an object may include, for example, priority information and spread information of the object.
[0521] The bitstream encoder 222 functions as a generator that generates a bitstream by appropriately encoding the configuration information, the metadata for each CVP of each object, and the audio data of each object supplied from the acquisition unit 221. The bitstream encoder 222 supplies the generated bitstream to the communication unit 223.
[0522] The configuration information may be generated for each frame of the content, for each section consisting of multiple frames, or for the entire content. The configuration information may also be stored in the bitstream as needed.
[0523] The communication unit 223 transmits the bit stream supplied from the bit stream encoder 222 to a client, which is an information processing device on the content playback side.
[0524] Note that, here, an example will be described in which the configuration information and the audio data and metadata of the object are transmitted to the client by one server 211. However, this is not limiting, and the configuration information and the audio data and metadata of the object may be transmitted to the client by different servers.
[0525] <Description of Bitstream Transmission Processing> Bitstream transmission processing by the server 211 will be described with reference to the flowchart of FIG.
[0526] In step S 341 , the acquisition unit 221 acquires information necessary for generating a bitstream and supplies it to the bitstream encoder 222 .
[0527] For example, the acquisition unit 221 acquires necessary information by receiving configuration information, metadata for each CVP of each object, audio data of each object, and the like transmitted from the information processing device 161 .
[0528] In step S342, the bitstream encoder 222 generates a bitstream by encoding and multiplexing the configuration information supplied from the acquisition unit 221, the metadata for each CVP of each object, and the audio data of each object, as appropriate, and supplies the bitstream to the communication unit 223. This generates a bitstream that includes the encoded configuration information, object metadata, object audio data, etc. as appropriate.
[0529] In step S343, the communication unit 223 transmits the bitstream supplied from the bitstream encoder 222 to the client, and the bitstream transmission process ends.
[0530] If one piece of configuration information is prepared for the entire content, the bitstream encoder 222 stores the configuration information in the beginning (header) portion of the bitstream in step S342.
[0531] Furthermore, if a plurality of different pieces of configuration information are prepared, for example, for each viewing area, the bitstream encoder 222 selects an appropriate piece of configuration information from the plurality of pieces of configuration information in step S342, and generates a bitstream including the selected piece of configuration information. In this case, as described with reference to Fig. 26, for example, at the timing when the configuration information is switched, the selected piece of configuration information is stored in the bitstream.
[0532] As an example, when configuration information is prepared for each viewing area, the bitstream encoder 222 acquires listener position information from the client via the communication unit 223. Then, when the viewing area including the listener position indicated by the listener position information changes, the bitstream encoder 222 stores the configuration information prepared for the viewing area after the change in the bitstream. In other words, the configuration information prepared for the viewing area to which the listener has moved is stored in the bitstream.
[0533] Similarly, for example, if configuration information is prepared for each client resource or each remaining battery level, the bitstream encoder 222 acquires resource information, which is information about the client's resources, and remaining battery level information, which indicates the remaining battery level, from the client via the communication unit 223. Then, the bitstream encoder 222 selects configuration information determined for the resource information or remaining battery level information, and stores the newly selected configuration information in the bitstream when the selected configuration information changes from the previous selection result. The resources referred to here may be the client's currently available resources (computational resources) or the maximum resources available to the client.
[0534] For example, when configuration information is prepared for each device type of a client, the bitstream encoder 222 acquires device type information, which is information indicating the device type of the client, from the client via the communication unit 223. Then, the bitstream encoder 222 selects configuration information determined for the device type information and stores the selected configuration information in the bitstream.
[0535] Furthermore, for example, if configuration information is prepared for each network state, the bitstream encoder 222 acquires the state of congestion and other information of the network that transmits the bitstream (content data) from the communication unit 223. Then, the bitstream encoder 222 selects configuration information determined for the network state, and stores the newly selected configuration information in the bitstream when the selected configuration information changes from the previous selection result.
[0536] Furthermore, for example, if configuration information is prepared for each scene of the content, the bitstream encoder 222 stores, in the bitstream, the configuration information determined for the scene after the scene change at the timing when the scene of the content changes.
[0537] In addition, the configuration information to be transmitted to the client may be selected from multiple pieces of configuration information based on a combination of at least two or more of the viewing area in which the listener is located, the client's resources, the client's remaining battery power, the client's device type, the network status, and the scene in which the content is being played.
[0538] Alternatively, the bitstream encoder 222 may store a plurality of different pieces of configuration information, such as for each viewing area, in the bitstream, so that the appropriate piece of configuration information can be selected on the client side.
[0539] In this way, the server 211 stores appropriate configuration information in the bitstream and transmits it to the client, thereby enabling the client receiving the bitstream to use the appropriate configuration information to control loudness by only controlling the gain of the object.
[0540] <Configuration Example of Client> FIG. 35 is a diagram showing a configuration example of an embodiment of a client to which the present technology is applied.
[0541] The client 261 shown in Figure 35 is an information processing device such as a personal computer, tablet terminal, smartphone, head-mounted device, or game device, and functions as a decoder that receives the bitstream transmitted by the server 211 and plays back the content.
[0542] The client 261 includes an input unit 271 , a display unit 272 , a communication unit 273 , a control unit 274 , a recording unit 275 , and an audio output unit 276 .
[0543] The input unit 271 includes, for example, a mouse, a keyboard, buttons, switches, and a touch panel superimposed on the display unit 272, and supplies signals according to operations by a user who is a listener to the control unit 274. The display unit 272 includes a display, and displays various images (display screens) such as images related to content under the control of the control unit 274.
[0544] The communication unit 273 communicates with an external device under the control of the control unit 274. For example, the communication unit 273 transmits listener position information supplied from the control unit 274 to the server 211, receives a bit stream transmitted from the server 211, and supplies the bit stream to the control unit 274.
[0545] The control unit 274 controls the overall operation of the client 261. For example, the control unit 274 executes an application program that performs processing related to the playback of content, thereby realizing the function of a decoder.
[0546] The recording unit 275 is made up of a non-volatile memory or the like, and records various data such as application programs for playing content, and supplies the recorded data to the control unit 274 as appropriate. The audio output unit 276 is made up of a speaker or the like, and outputs sound based on audio data supplied from the control unit 274. Note that the audio output unit 276 may be provided outside the client 261. In such a case, for example, headphones, earphones, hearing aids, or the like may be used as the audio output unit 276.
[0547] <Example of Functional Configuration of Client> The control unit 274 of the client 261 executes an application program to realize the function of a decoder.
[0548] Fig. 36 is a diagram showing an example of a functional configuration in which the client 261 functions as a decoder. In Fig. 36, parts corresponding to those in Fig. 35 are given the same reference numerals, and their explanation will be omitted as appropriate.
[0549] The client 261 shown in FIG. 36 includes a communication unit 273 , a bitstream decoder 301 , a metadecoder 302 , a rendering processing unit 303 , and an audio output unit 276 .
[0550] For example, the bitstream decoder 301, metadecoder 302, and rendering processing unit 303 are realized by the control unit 274 executing an application program. The bitstream decoder 301 and metadecoder 302 correspond to the 6DoF decoder 121 shown in Fig. 23, and the rendering processing unit 303 corresponds to the rendering module 122 shown in Fig. 23.
[0551] The communication unit 273 receives the bitstream transmitted from the server 211 and supplies it to the bitstream decoder 301. In other words, the communication unit 273 functions as an acquisition unit that receives the bitstream and acquires the configuration information and metadata contained in the bitstream.
[0552] The bitstream decoder 301 functions as a decoding unit that decodes the bitstream supplied from the communication unit 273, more specifically, the encoded audio data and the like included in the bitstream. Through decoding and the like in the bitstream decoder 301, configuration information, metadata for each CVP of each object, and audio data of each object are extracted from the bitstream.
[0553] The bitstream decoder 301 supplies the audio data of each object to the rendering processing unit 303, and also supplies the configuration information and metadata for each CVP of each object to the meta decoder 302.
[0554] The meta decoder 302 is supplied with listener position information indicating the absolute position of the listener in the three-dimensional virtual space in which the object is placed, and listener direction information indicating the orientation of the listener in the three-dimensional virtual space, from the higher-level control unit 274 as appropriate.
[0555] The meta decoder 302 generates listener-based metadata, which is metadata for each object based on the listener position, based on the configuration information supplied from the bitstream decoder 301, the metadata for each CVP of each object, the listener position information, and the listener direction information, and supplies this to the rendering processing unit 303.
[0556] The listener-based metadata includes listener-based object position information, which indicates the position of each object relative to the listener position, and listener-based gain information, which is gain information of the object relative to the listener position. In addition, the listener-based metadata may include priority information and spread information of each object.
[0557] For example, the listener-reference object position information is information that indicates the relative position of an object as seen from the listener, expressed using coordinates (polar coordinates) in a polar coordinate system with the listener's position in virtual space as the reference (origin).
[0558] When generating listener-reference metadata, the gain information for each CVP of an object is corrected to generate corrected gain information, and the listener-reference gain information is generated based on the corrected gain information of the object for each CVP. Therefore, the meta decoder 302 can be said to function as a correction unit that corrects the gain information of an object according to the loudness mode.
[0559] The rendering processing unit 303 performs rendering processing based on the audio data of each object supplied from the bitstream decoder 301 and the listener-reference metadata supplied from the meta decoder 302, and generates output audio data of the content.
[0560] The rendering processing unit 303 generates output audio data by performing rendering processing in a polar coordinate system defined by MPEG-H, such as VBAP. Note that the rendering processing is not limited to VBAP and may be any other processing. For example, BRIR, HRTF, HOA, ITD (Interaural Time Difference), IID (Interaural Intensity Difference), etc. may be used in the rendering processing.
[0561] The output audio data of the content is made up of audio data for each channel that is supplied to speakers corresponding to each channel that constitute a speaker system serving as the audio output unit 276, for example.
[0562] The rendering processing unit 303 outputs the output audio data of the content to the sound output unit 276, thereby causing the sound output unit 276 to play back the sound of the content including the sounds of all objects. At this time, the sound (sound image) of each object is localized at the position indicated by the listener reference object position information.
[0563] <Description of Output Audio Data Generation Processing> The output audio data generation processing by the client 261 will be described with reference to the flowchart of FIG.
[0564] In step S391, the communication unit 273 receives the bitstream transmitted from the server 211 in step S343 of FIG.
[0565] In step S392, the bitstream decoder 301 decodes the encoded audio data and the like contained in the bitstream supplied from the communication unit 273, and extracts various pieces of information contained in the bitstream, thereby extracting configuration information, metadata for each CVP of each object, and audio data for each object.
[0566] The bitstream decoder 301 supplies the audio data of each object to the rendering processing unit 303, and also supplies the configuration information and metadata for each CVP of each object to the meta decoder 302.
[0567] In step S393, the meta decoder 302 generates listener-reference metadata based on the configuration information supplied from the bitstream decoder 301, the metadata for each CVP of each object, and the supplied listener position information and listener direction information.
[0568] The meta decoder 302 supplies the generated listener-based metadata to the rendering processing unit 303. The meta decoder 302 generates the listener-based metadata as will be described later in detail.
[0569] Furthermore, a plurality of different pieces of configuration information may be included in the bitstream received (acquired) by the communication unit 273. In such a case, when generating the listener-based metadata, the meta decoder 302 may select one piece of configuration information from the plurality of pieces of configuration information, and generate the listener-based metadata using the selected piece of configuration information.
[0570] The selection of configuration information may be performed only when playback of the content starts, or may be performed for each frame of the content, or may be performed for each fixed or variable period, such as each period consisting of multiple frames. When the selection of configuration information is performed multiple times, the configuration information used to generate the listener-based metadata is switched appropriately depending on the position of the listener, the computing resources, etc.
[0571] As an example, if configuration information is prepared for each viewing area in a virtual space, the meta decoder 302 appropriately uses the listener position information to identify the viewing area in which the listener is currently located by some means, and selects the configuration information prepared for the identified viewing area.
[0572] For example, the viewing area may be specified by the meta decoder 302 transmitting listener position information to the server 211 via the communication unit 273, and then acquiring, via the communication unit 273, information indicating the viewing area including the current listener position, which information is transmitted from the server 211 in response to the transmission. Alternatively, the meta decoder 302 may previously store information indicating the range of each viewing area in virtual space, and specify the viewing area including the current listener position based on that information and the listener position information. Alternatively, information indicating the viewing area including the current listener position may be stored in the bitstream.
[0573] Similarly, if configuration information is prepared for each resource or remaining battery level of the client 261, the meta decoder 302 acquires resource information indicating the currently available resources (computational resources) of the client 261 or the maximum resources available to the client 261, and remaining battery level information indicating the remaining battery level of the client 261. The meta decoder 302 then selects configuration information determined for the resource or remaining battery level indicated by the resource information or remaining battery level information.
[0574] For example, if configuration information is prepared for each device type of client 261, meta decoder 302 identifies the device type of client 261 by some means, such as by referring to device type information previously recorded in client 261, and selects the configuration information defined for the identified device type.
[0575] Also, for example, if configuration information is prepared for each network state, the meta decoder 302 obtains from the communication unit 273 the state of the network, such as the congestion state, through which the bit stream (content data) is transmitted, and selects the configuration information defined for the network state.
[0576] Furthermore, for example, if configuration information is prepared for each scene of the content, the meta decoder 302 identifies the scene currently being played back in the content, more specifically, the scene that is about to be played back, by some means, and selects the configuration information determined for the identified scene. For example, information indicating the scene to be played back in each frame of the content may be stored in the bitstream, or information indicating the scene to be played back at each time (frame) of the content may be stored in advance in the meta decoder 302.
[0577] Alternatively, configuration information designated by the listener may be selected from among a plurality of pieces of configuration information by operating the input unit 271. In other words, the configuration information selected by the listener may be used in accordance with the listener's operation.
[0578] In addition, the configuration information used to generate the listener-reference metadata may be selected based on a combination of at least two of the viewing area where the listener is located, the resources of the client 261, the remaining battery level of the client 261, the device type of the client 261, the network status, and the scene during playback of the content.
[0579] In step S 394 , the rendering processing unit 303 performs rendering processing based on the audio data of each object supplied from the bitstream decoder 301 and the listener-reference metadata supplied from the meta decoder 302 .
[0580] For example, in the rendering process, audio data for each channel is generated for each object to play the sound of that object, and the audio data for the same channel obtained for each object is added together to generate output audio data for each channel of the content.
[0581] The rendering processing unit 303 supplies the output audio data for each channel obtained by the rendering processing to the audio output unit 276. The audio output unit 276 plays back the sound of the content based on the output audio data supplied from the rendering processing unit 303. When the sound of the content has been played back, the output audio data generation processing ends.
[0582] In this way, the client 261 generates listener-reference metadata using the configuration information, and generates output audio data by rendering based on the listener-reference metadata. This allows for appropriate loudness control and free viewpoint audio playback that reflects the intentions of the producer and the target loudness value specified by the listener.
[0583] <Explanation of Metadata Generation Process> The client 261 performs the metadata generation process shown in FIG. 38 as part of the process of step S393 in the output audio data generation process described with reference to FIG.
[0584] The metadata generation process by the client 261 will be described below with reference to the flowchart of FIG.
[0585] In step S441, the meta decoder 302 determines whether or not the mode is the loudness mode.
[0586] For example, when playing back content, the control unit 274 causes the display unit 272 to display the display screen shown in Fig. 30. When this display screen is displayed and the listener (user) appropriately operates the input unit 271 to cause a check mark to appear in the check box BX11, i.e., when a check mark is displayed in the check box BX11, the meta decoder 302 determines that the loudness mode is in effect. Note that the loudness mode may also be determined to be in effect when a target loudness value is input in the input field IPB11.
[0587] If it is determined in step S441 that the mode is not the loudness mode, then the process proceeds to step S454. In this case, the gain information of the object included in the metadata for each CVP of the object is used as the correction gain information as is. In other words, the gain information is not corrected.
[0588] On the other hand, if it is determined in step S441 that the mode is the loudness mode, the meta decoder 302 determines in step S442 whether or not the configuration information supplied from the bit stream decoder 301 stores a measured loudness value.
[0589] If it is determined in step S442 that a measured loudness value is not stored, processing in loudness mode cannot be performed, and the process then proceeds to step S454. In this case, as in the case where it is determined in step S441 that the loudness mode is not set, the gain information of the object is not corrected, and the gain information is used as corrected gain information as is.
[0590] On the other hand, if it is determined in step S442 that a measured loudness value is stored, the meta decoder 302 determines in step S443 whether or not the mode is the measured loudness mode based on the configuration information.
[0591] For example, when the configuration information includes the multi-loudness information shown in Fig. 24, the measured loudness mode is determined when the value of the group mode flag information included in the configuration information is "0" and the value of the production loudness value presence flag information included in the configuration information is "0." More specifically, since the production loudness value presence flag information is stored when the value of the group mode flag information is "0," the measured loudness mode is determined when the value of the production loudness value presence flag information is "0."
[0592] If it is determined in step S443 that the mode is the measured loudness mode, the meta decoder 302 calculates a loudness change value for each CVP in step S444 based on the measured loudness value of the CVP and the target loudness value specified by the listener. For example, the meta decoder 302 calculates the loudness change value for each CVP by performing a calculation similar to that of equation (11) above using the measured loudness value included in the configuration information.
[0593] In step S445, the meta decoder 302 corrects the gain information included in the metadata for each CVP of each object supplied from the bitstream decoder 301 based on the loudness modification value for each CVP.
[0594] For example, for each object, for each CVP, the meta decoder 302 performs calculations similar to the above-mentioned equations (12) and (13) based on the gain information of the object contained in the metadata and the loudness change value calculated in step S444.
[0595] The gain change rate for each CVP is calculated from the loudness change value using a calculation similar to that of Equation (12). Furthermore, the gain information for each object is corrected by multiplying the gain change rate by the gain information for each CVP of the object using a calculation similar to that of Equation (13). This results in corrected gain information, which is the corrected gain information.
[0596] In this case, the gain information of the object defined for the CVP is corrected so that the loudness of the output audio data when the CVP is positioned as the listener's position becomes the target loudness value.
[0597] Once the gain information for each CVP of each object has been corrected, the process proceeds to step S454. In this case, the metadata for each CVP of the object includes at least the correction gain information and object position information of the object.
[0598] If it is determined in step S443 that the mode is not the measurement loudness mode, the meta decoder 302 determines in step S446 whether the mode is the group mode or not based on the configuration information.
[0599] For example, if the multi-loudness information shown in Figure 24 is included in the configuration information, it is determined to be in group mode when the value of the group mode flag information included in the configuration information is "1" or "2".
[0600] If it is determined in step S446 that the mode is not the group mode, then the process proceeds to step S447.
[0601] In this case, for example, when the multi-loudness information shown in Figure 24 is included in the configuration information, the value of the group mode flag information is "0" and the value of the production loudness value presence flag information is "1", so the production loudness mode is selected.
[0602] In step S447, the meta-decoder 302 calculates, for each CVP, an intermediate loudness modification value based on the measured loudness value of the CVP and the production loudness value of the CVP.
[0603] For example, the meta decoder 302 calculates the intermediate loudness change value for each CVP by performing a calculation similar to that of equation (17) above using the measured loudness value and the production loudness value included in the configuration information.
[0604] In step S448, the meta decoder 302 calculates a common correction amount based on the production loudness value and the target loudness value.
[0605] For example, the meta decoder 302 determines the maximum of the production loudness values of all CVPs as the maximum production loudness value, and calculates the common correction amount by performing a calculation similar to the above-mentioned equation (18) based on that maximum production loudness value and the target loudness value.
[0606] In step S449, the meta decoder 302 calculates a final loudness change value for each CVP based on the intermediate loudness change value calculated in step S447 and the common correction amount calculated in step S448. For example, the meta decoder 302 calculates the final loudness change value for each CVP by performing a calculation similar to that of equation (19) above.
[0607] In step S450, the meta decoder 302 corrects the gain information included in the metadata for each CVP of each object supplied from the bitstream decoder 301 based on the final loudness change value for each CVP.
[0608] For example, for each object, for each CVP, the meta decoder 302 performs calculations similar to the above-mentioned equations (20) and (21) based on the gain information of the object contained in the metadata and the final loudness change value calculated in step S449.
[0609] The gain change rate for each CVP is calculated from the final loudness change value using a calculation similar to that of equation (20). Also, for each object, the gain change rate is multiplied by the gain information for each CVP of the object using a calculation similar to that of equation (21) to obtain corrected gain information.
[0610] Once the gain information for each CVP of each object has been corrected, the process proceeds to step S454. In this case, the metadata for each CVP of the object includes at least the correction gain information and object position information of the object.
[0611] In this case, the loudness of the output audio data when the CVP with the maximum production loudness value is positioned as the listener becomes the target loudness value, and the gain information is corrected so that the relative relationship between the loudness of the output audio data at each of the multiple CVPs becomes the same as the relative relationship between the production loudness values at each of the multiple CVPs.
[0612] If it is determined in step S446 that the mode is the group mode, the meta decoder 302 identifies the maximum measured loudness value for each group (CVP group) in step S451.
[0613] For example, the metadecoder 302 identifies the CVP group to which each CVP belongs based on the group index value of each CVP included in the configuration information. The metadecoder 302 also identifies the maximum measured loudness value, which is the maximum value among the measured loudness values of the CVPs belonging to the CVP group, based on the measured loudness values of each CVP included in the configuration information and the result of identifying the CVP group. When the value of the group mode flag information is "2," all CVPs are considered to belong to the same group.
[0614] In step S452, the metadecoder 302 calculates a loudness change value for each CVP for each CVP group based on the maximum measured loudness value identified in step S451 and the target loudness value. For example, the metadecoder 302 determines the loudness change value to be a value (difference value) obtained by subtracting the maximum measured loudness value from the target loudness value. In step S452, the loudness change value for each CVP group is found (calculated).
[0615] In step S453, the meta decoder 302 corrects the gain information included in the metadata for each CVP of each object supplied from the bitstream decoder 301 based on the loudness modification value for each CVP.
[0616] For example, for each object, for each CVP, the meta decoder 302 performs calculations similar to the above-mentioned equations (15) and (16) based on the gain information of the object contained in the metadata and the loudness change value calculated in step S452.
[0617] The gain change rate for each CVP is calculated from the loudness change value using a calculation similar to that of equation (15). Also, for each object, the gain change rate is multiplied by the gain information for each CVP of the object using a calculation similar to that of equation (16) to obtain corrected gain information.
[0618] In this case, the loudness of the output audio data when the listener is positioned at the CVP with the largest measured loudness value among the CVPs belonging to the same group becomes the target loudness value, and the gain information is corrected so that the relative relationship between the loudness of the output audio data at each of the multiple CVPs belonging to the same group becomes the same as the relative relationship between the measured loudness values of each of the multiple CVPs.
[0619] Once the gain information for each CVP of each object has been corrected, the process proceeds to step S454. In this case, the metadata for each CVP of the object includes at least the correction gain information and object position information of the object.
[0620] If it is determined in step S441 that the loudness mode is not active, if it is determined in step S442 that the measured loudness value is not stored, if the processing of step S445 is performed, if the processing of step S450 is performed, or if the processing of step S453 is performed, then the processing of step S454 is performed.
[0621] In step S454, the meta decoder 302 performs an interpolation process to generate listener-based metadata for each object, that is, listener-based metadata.
[0622] For example, let us consider one object.
[0623] The metadecoder 302 selects a CVP to be used for interpolation based on the listener position information and the CVP position information included in the configuration information. The CVPs to be used for interpolation may be a portion of all CVPs located around the listener position, or all CVPs may be used for interpolation.
[0624] Next, the metadecoder 302 calculates a weighting factor for each CVP based on the CVP position information and the listener position information. For example, the weighting factor for each CVP is determined by the reciprocal ratio of the distance from the CVP to the listener position.
[0625] Furthermore, based on the object position information for each CVP, the meta decoder 302 calculates a three-dimensional object position vector that has the CVP as its start point and the position of the object as seen from the CVP as its end point.
[0626] The metadecoder 302 performs interpolation to calculate the weighted vector sum of the object three-dimensional position vectors for each CVP, using the weighting coefficients calculated for each CVP as weights, and uses the resulting vector (position information) as listener-reference object position information. In other words, the sum of the object three-dimensional position vectors for each CVP multiplied by the weighting coefficients is calculated as the listener-reference object position information.
[0627] The object three-dimensional position vector obtained by the above calculation is an absolute coordinate in an absolute coordinate system with the listener position as the origin. However, when rendering processing is performed in a polar coordinate system by the rendering processing unit 303, listener-reference object position information expressed in polar coordinates is required. Therefore, the meta decoder 302 appropriately converts the listener-reference object position information expressed in absolute coordinates into listener-reference object position information expressed in polar coordinates.
[0628] Furthermore, listener direction information may be used as appropriate to calculate listener reference object position information.
[0629] The meta decoder 302 calculates listener reference gain information, which is gain information of the object relative to the listener position, by interpolation based on the correction gain information included in the metadata for each CVP of the object. The listener reference gain information is gain information of the object when the listener position is the listening position.
[0630] As an example, the metadecoder 302 multiplies the correction gain information of each CVP by the weighting coefficient calculated for each CVP, and sets the sum of the correction gain information multiplied by the weighting coefficient as the listener reference gain information.
[0631] As a specific calculation method for obtaining the listener-reference object position information and the listener-reference gain information by interpolation, for example, the method described in International Publication No. 2023 / 085140 can be adopted. The calculation method for the listener-reference object position information and the listener-reference gain information is not limited to this, and any other method may be used. For example, the CVP closest to the listener position may be identified, and the metadata of the object for the identified CVP may be used as the listener-reference metadata.
[0632] The meta decoder 302 calculates metadata including at least listener-based object position information and listener-based gain information as listener-based metadata for the object.
[0633] If the listener-reference metadata includes priority information, spread information, etc., the priority information and spread information may also be generated by interpolation or the like.
[0634] Also, for example, the priority information and spread information included in the metadata for each CVP of the object used to generate the listener-based metadata may be the highest or lowest value, or the value for the CVP closest to the listener position, etc. Furthermore, the median or average value of the priority information and spread information may be used as the priority information and spread information for the listener-based metadata.
[0635] When the meta decoder 302 supplies the listener-based metadata to the rendering processing unit 303, the metadata generation processing is completed.
[0636] In the subsequent rendering processing unit 303, rendering processing is performed based on the listener-reference metadata and audio data of each object in step S394 of FIG.
[0637] For example, the rendering processing unit 303 performs gain correction on the audio data of each object based on the listener reference gain information of that object. Then, the rendering processing unit 303 performs VBAP or the like based on the gain-corrected audio data of each object and the listener reference object position information of each object, thereby generating output audio data.
[0638] By using such listener reference gain information, the client 261 can achieve loudness correction in free viewpoint audio by only controlling the gain of the object (gain correction).
[0639] In this way, the client 261 generates listener-reference metadata based on the configuration information. In this way, loudness control can be performed using only gain control, and free viewpoint audio playback can be realized that reflects the intentions of the producer and the target loudness value specified by the listener.
[0640] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.
[0641] FIG. 39 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.
[0642] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .
[0643] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.
[0644] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0645] In a computer configured as described above, the CPU 501 loads a program recorded in the recording unit 508, for example, into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the above-described series of processes.
[0646] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0647] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.
[0648] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.
[0649] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.
[0650] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.
[0651] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.
[0652] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.
[0653] Furthermore, the present technology can also be configured as follows.
[0654] (1) An information processing device comprising: an acquisition unit that acquires loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed; and a level correction unit that performs level correction on audio data of the object based on the loudness information for each of the plurality of positions or orientations. (2) The information processing device described in (1), wherein the acquisition unit acquires the loudness information for each of a plurality of orientations based on a control viewpoint defined for each of the control viewpoints in the space. (3) The information processing device described in (2), wherein the placement position of the object in the space differs for each of the control viewpoints. (4) The information processing device described in any one of (1) to (3), further comprising a loudness information processing unit that calculates an attenuation coefficient based on all of the loudness information for each position or orientation, wherein the level correction unit performs the level correction on the audio data based on the attenuation coefficient. (5) The information processing device described in (4), wherein the loudness information processing unit calculates the attenuation coefficient based on the loudness information with the largest value among all of the loudness information and the maximum level of sound that can be recorded as digital audio data. (6) The information processing device according to (2) or (3), further comprising a loudness information processing unit that selects, based on listener orientation information indicating an orientation of the listener in the space, from the loudness information for each of a plurality of orientations at a predetermined control viewpoint, the loudness information for an orientation closest to an orientation indicated by the listener orientation information, and the level correction unit performs the level correction of the audio data based on the loudness information selected by the loudness information processing unit. (7) The information processing device according to (6), further comprising: a loudness information processing unit that selects, as the predetermined control viewpoint, the control viewpoint closest to a position of the listener based on listener position information indicating a position of the listener in the space.(8) The information processing device according to (2) or (3), further comprising a loudness information processing unit that selects, for each of the plurality of control viewpoints, from the loudness information for each of a plurality of orientations at the control viewpoint based on listener orientation information indicating an orientation of the listener in the space, the loudness information for an orientation closest to an orientation indicated by the listener orientation information, and performs interpolation processing based on the plurality of pieces of loudness information selected for each of the control viewpoints, wherein the level correction unit performs the level correction of the audio data based on the loudness information obtained by the interpolation processing. (9) The information processing device according to (8), wherein the loudness information processing unit performs the interpolation processing based on a ratio of distances from the control viewpoints to positions of the listeners in the space. (10) The information processing device according to any one of (6) to (9), wherein the level correction unit performs the level correction by DRC processing based on the loudness information. (11) The information processing device according to any one of (1) to (10), further comprising a rendering processing unit that performs rendering processing based on the audio data of the object and metadata of the object, wherein the level correction unit performs the level correction on an output signal obtained by the rendering processing. (12) The information processing device according to (11), wherein the metadata is at least one of position information of the object, gain information of the object, priority information of the object, and spread information of the object. (13) The information processing device according to (11) or (12), wherein the rendering processing is processing using at least one of VBAP, BRIR, HRTF, and HOA. (14) The information processing device according to any one of (1) to (13), wherein the loudness information is a sample peak level value or a true peak level value. (15) An information processing method, in which an information processing device acquires loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and performs level correction on audio data of the object based on the loudness information for each of the plurality of positions or orientations.(16) A program causing a computer to execute a process including the steps of acquiring loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and performing level correction on audio data of the object based on the loudness information for each of the plurality of positions or orientations. (17) An information processing device comprising: a generation unit that generates a bitstream including loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and a communication unit that transmits the bitstream. (18) The information processing device according to (17), wherein the generation unit generates the bitstream including the loudness information for each of a plurality of orientations based on the control viewpoint, which is defined for each control viewpoint in the space. (19) The information processing device according to (18), wherein the generation unit generates the bitstream including metadata including object position information indicating the position of the object and the loudness information. (20) The information processing device according to (19), wherein the placement position of the object in the space differs for each of the control viewpoints. (21) The information processing device according to any one of (18) to (20), wherein the generation unit generates the bitstream including the loudness information and system configuration information including control viewpoint position information indicating the position of the control viewpoint in the space. (22) The information processing device according to any one of (17) to (21), wherein the generation unit generates the bitstream including the loudness information and audio data of the object. (23) An information processing method, wherein an information processing device generates a bitstream including loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and transmits the bitstream. (24) A program that causes a computer to execute processing including the steps of generating a bitstream including loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and transmitting the bitstream.(25) An information processing device comprising: a correction unit that corrects gain information of an object determined for a control viewpoint in space based on a measured loudness value that is a measurement result of loudness of audio data of content including sounds of one or more objects when the control viewpoint is set to the position of a listener, and a predetermined target loudness value. (26) The information processing device described in (25), wherein the correction unit performs the correction so that the loudness of the audio data of the content when the control viewpoint is set to the position of the listener becomes the target loudness value. (27) The information processing device described in (25), wherein the correction unit corrects the gain information of each of a plurality of control viewpoints based on any of the measured loudness values of the plurality of control viewpoints and the target loudness value. (28) The information processing device described in (27), wherein the correction unit corrects the gain information of each of the control viewpoints belonging to the same group based on a change value calculated from the target loudness value and the maximum value of the measured loudness values of the plurality of control viewpoints belonging to the same group. (29) The information processing device according to (28), wherein the correction unit corrects the gain information so that the loudness of the audio data of the content becomes the target loudness value when the control viewpoint at which the measured loudness value is the maximum is set as the position of the listener. (30) The information processing device according to (25), wherein the correction unit corrects the gain information of each of the plurality of control viewpoints based on a production loudness value specified for each of the control viewpoints, the measured loudness value of the control viewpoint, and the target loudness value. (31) The information processing device according to (30), wherein the correction unit calculates a common correction amount based on the production loudness value of each of the plurality of control viewpoints and the target loudness value, and corrects the gain information of the control viewpoint based on the common correction amount, the production loudness value of the control viewpoint, and the measured loudness value of the control viewpoint. (32) The information processing device according to (31), wherein the correction unit calculates the common correction amount based on a maximum value among the production loudness values of the plurality of control viewpoints and the target loudness value.(33) The information processing device according to (32), wherein the correction unit corrects the gain information so that the loudness of the audio data of the content becomes the target loudness value when the control viewpoint at which the production loudness value is the maximum is positioned as the listener. (34) The information processing device according to any one of (25) to (33), further comprising an acquisition unit that acquires configuration information including the measured loudness value of the control viewpoint. (35) The information processing device according to (34), wherein the configuration information includes information for specifying whether to correct the gain information based on the measured loudness value and the target loudness value, to correct the gain information based on the measured loudness value, information indicating a group to which the control viewpoint belongs that is included in the configuration information, and the target loudness value, or to correct the gain information based on the measured loudness value, a production loudness value specified for each control viewpoint that is included in the configuration information, and the target loudness value. (36) The information processing device according to (34) or (35), wherein the acquisition unit acquires a plurality of pieces of the configuration information, and the correction unit corrects the gain information using one piece of the configuration information selected from the plurality of pieces of configuration information. (37) The information processing device according to (36), wherein the correction unit selects the configuration information in response to an operation of the listener, or selects the configuration information based on at least one of a position of the listener in the space, resources of the information processing device, a remaining battery level of the information processing device, a device type of the information processing device, a state of a network over which data of the content is transmitted, and a scene of the content. (38) The information processing device according to any one of (25) to (37), wherein the correction unit calculates listener reference gain information, which is the gain information of the object with respect to the position of the listener, based on the gain information after correction of the plurality of control viewpoints, position information indicating positions of the plurality of control viewpoints in the space, and listener position information indicating the position of the listener in the space.(39) The information processing device according to (38), further comprising a rendering processing unit that generates audio data of the content based on the listener reference gain information of the object and audio data of the object. (40) The information processing device according to any one of (25) to (39), wherein the arrangement position of the object in the space differs for each of the control viewpoints. (41) The information processing device according to any one of (25) to (40), further comprising a control unit that displays an image of the space in which the object is arranged. (42) The information processing device according to (41), wherein the image displays an area for inputting the target loudness value. (43) An information processing method, wherein an information processing device corrects gain information of the object defined for the control viewpoint based on a measured loudness value that is a result of measuring loudness of audio data of content including sounds of one or more objects when a control viewpoint in the space is set as the position of the listener, and a predetermined target loudness value. (44) A program causing a computer to execute a process including the step of correcting gain information of an object defined for a control viewpoint in a space based on a measured loudness value that is a result of measuring the loudness of audio data of content including sounds of one or more objects when the control viewpoint in a space is set as the position of a listener, and a predetermined target loudness value. (45) An information processing device comprising: a control unit that generates configuration information including a measured loudness value that is a result of measuring the loudness of audio data of content including sounds of one or more objects when the control viewpoint in a space is set as the position of a listener. (46) The information processing device according to (45), wherein the configuration information includes group mode information that indicates whether or not a group mode is in effect for correcting gain information of the object defined for the control viewpoint belonging to the same group, based on any of the measured loudness values of the control viewpoints belonging to the same group and a predetermined target loudness value.(47) The information processing device according to (46), wherein, when the configuration information includes group mode information indicating the group mode, the configuration information further includes information indicating the group to which the control viewpoint belongs. (48) The information processing device according to any one of (45) to (47), wherein the configuration information includes production loudness value presence information indicating whether a production loudness value specified for each of the control viewpoints is included, and when the configuration information includes the production loudness value presence information indicating the inclusion of the production loudness value, the configuration information further includes the production loudness value of the control viewpoint. (49) The information processing device according to any one of (45) to (48), wherein the configuration information includes position information indicating the position of the control viewpoint in the space. (50) The information processing device according to any one of (45) to (49), wherein the control unit generates a plurality of different pieces of configuration information. (51) The information processing device according to (50), wherein the control unit generates the configuration information for each region in the space, for each resource of an information processing device that reproduces the content, for each remaining battery level of an information processing device that reproduces the content, for each device type of an information processing device that reproduces the content, for each state of a network through which data of the content is transmitted, or for each scene of the content. (52) The information processing device according to any one of (45) to (51), wherein the control unit displays an image of the space in which the control viewpoint is arranged, and the measured loudness value of the control viewpoint is displayed on the image. (53) The information processing device according to (52), wherein a user interface for specifying a group to which the control viewpoint belongs is displayed on the image. (54) The information processing device according to (52) or (53), wherein a production loudness value specified for each control viewpoint is displayed on the image. (55) The information processing device according to any one of (45) to (54), wherein an arrangement position of the object in the space differs for each control viewpoint.(56) An information processing method in which an information processing device generates configuration information including measured loudness values that are the measurement results of loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set as the position of a listener. (57) A program that causes a computer to execute processing including the step of generating configuration information including measured loudness values that are the measurement results of loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set as the position of a listener. (58) An information processing device comprising: a generation unit that generates a bitstream including configuration information storing measured loudness values that are the measurement results of loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set as the position of a listener; and a communication unit that transmits the bitstream. (59) The information processing device described in (58), in which the configuration information includes group mode information that indicates whether or not a group mode is in which gain information of the object defined for the control viewpoint belonging to the same group is corrected based on any of the measured loudness values of the control viewpoints belonging to the same group and a predetermined target loudness value. (60) The information processing device according to (59), wherein, when the configuration information includes group mode information indicating the group mode, the configuration information further includes information indicating the group to which the control viewpoint belongs. (61) The information processing device according to any one of (58) to (60), wherein the configuration information includes production loudness value presence information indicating whether a production loudness value specified for each of the control viewpoints is included, and when the configuration information includes the production loudness value presence information indicating the inclusion of the production loudness value, the configuration information further includes the production loudness value of the control viewpoint. (62) The information processing device according to any one of (58) to (61), wherein the configuration information includes position information indicating the position of the control viewpoint in the space.(63) The information processing device according to any one of (58) to (62), wherein the generation unit generates the bitstream including the configuration information selected from a plurality of different pieces of the configuration information. (64) The information processing device according to (63), wherein the generation unit selects the configuration information based on at least one of a position of the listener in the space, resources of the information processing device that reproduces the content, remaining battery levels of the information processing device that reproduces the content, a device type of the information processing device that reproduces the content, a state of a network through which data of the content is transmitted, and a scene of the content. (65) The information processing device according to any one of (58) to (62), wherein the generation unit generates the bitstream including a plurality of different pieces of the configuration information. (66) The information processing device according to any one of (58) to (65), wherein a placement position of the object in the space differs for each control viewpoint. (67) An information processing method in which an information processing device generates a bitstream including configuration information in which measured loudness values are stored as the loudness measurement results of audio data of content including sounds of one or more objects when a control viewpoint in a space is set as the position of a listener, and transmits the bitstream. (68) A program that causes a computer to execute processing including the steps of: generating a bitstream including configuration information in which measured loudness values are stored as the loudness measurement results of audio data of content including sounds of one or more objects when a control viewpoint in a space is set as the position of a listener, and transmitting the bitstream.
[0655] REFERENCE SIGNS LIST 11 Server, 21 Acquisition unit, 22 Bitstream encoder, 23 Communication unit, 51 Client, 61 Communication unit, 62 Bitstream decoder, 63 Metadecoder, 64 Rendering processing unit, 65 Loudness information processing unit, 66 DRC processing unit
Claims
1. An information processing device comprising: an acquisition unit that acquires loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed; and a level correction unit that performs level correction on the audio data of the object based on the loudness information for each of the plurality of positions or orientations.
2. The information processing device according to claim 1, wherein the acquisition unit acquires the loudness information for each of a plurality of directions based on a control viewpoint defined for each control viewpoint in the space.
3. The information processing device according to claim 2, wherein the object is arranged at a different position in the space for each of the control viewpoints.
4. The information processing device according to claim 1, further comprising a loudness information processing unit that calculates an attenuation coefficient based on all of the loudness information for each position or orientation, wherein the level correction unit performs the level correction of the audio data based on the attenuation coefficient.
5. The information processing device according to claim 4, wherein the loudness information processing section calculates the attenuation coefficient based on the loudness information having the largest value among all the loudness information and the maximum sound level that can be recorded as digital audio data.
6. The information processing device of claim 2, further comprising a loudness information processing unit that selects, from the loudness information for multiple orientations at a specific control viewpoint based on listener orientation information indicating the orientation of the listener in the space, the loudness information for an orientation closest to the orientation indicated by the listener orientation information, and the level correction unit performs the level correction of the audio data based on the loudness information selected by the loudness information processing unit.
7. The information processing device according to claim 6, wherein the loudness information processing section selects the control viewpoint closest to the position of the listener as the predetermined control viewpoint based on listener position information indicating the position of the listener in the space.
8. The information processing device of claim 2, further comprising a loudness information processing unit that selects, for each of a plurality of control viewpoints, from the loudness information for each of a plurality of orientations at the control viewpoint based on listener orientation information indicating the orientation of the listener in the space, the loudness information for a direction closest to the orientation indicated by the listener orientation information, and performs an interpolation process based on the plurality of loudness information selected for each of the control viewpoints, and the level correction unit performs the level correction of the audio data based on the loudness information obtained by the interpolation process.
9. The information processing device according to claim 8, wherein the loudness information processing section performs the interpolation process based on a ratio of a distance from the control viewpoint to the position of the listener in the space.
10. The information processing device according to claim 6, wherein the level correction section performs the level correction by DRC processing based on the loudness information.
11. The information processing device according to claim 1, further comprising a rendering processing unit that performs rendering processing based on the audio data of the object and metadata of the object, wherein the level correction unit performs the level correction on an output signal obtained by the rendering processing.
12. The information processing device according to claim 11, wherein the metadata is at least one of position information of the object, gain information of the object, priority information of the object, and spread information of the object.
13. The information processing device according to claim 11, wherein the rendering process uses at least one of VBAP, BRIR, HRTF, and HOA.
14. The information processing device according to claim 1, wherein the loudness information is a sample peak level value or a true peak level value.
15. An information processing method, in which an information processing device acquires loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed, and performs level correction on the audio data of the object based on the loudness information for each of the plurality of positions or orientations.
16. An information processing device comprising: a generation unit that generates a bit stream including loudness information defined for each of a plurality of positions or orientations that a listener can take in a space in which an object is placed; and a communication unit that transmits the bit stream.
17. An information processing device comprising: a correction unit that corrects gain information of one or more objects, determined for a control viewpoint in space, based on a measured loudness value, which is a measurement result of the loudness of audio data of content including sounds of one or more objects when the control viewpoint is set to the position of a listener, and a predetermined target loudness value.
18. The information processing device according to claim 17, wherein the correction unit performs the correction so that the loudness of the audio data of the content when the control viewpoint is set to the position of the listener becomes the target loudness value.
19. The information processing device according to claim 17, wherein the correction unit corrects the gain information for each of the plurality of control viewpoints based on any one of the measured loudness values for the plurality of control viewpoints and the target loudness value.
20. The information processing device according to claim 19, wherein the correction unit corrects the gain information of each of the control viewpoints belonging to the same group based on a modification value calculated from the maximum value of the measured loudness values of the control viewpoints belonging to the same group and the target loudness value.
21. The information processing device according to claim 20, wherein the correction unit corrects the gain information so that the loudness of the audio data of the content when the control viewpoint at which the measured loudness value is the maximum value is set as the position of the listener becomes the target loudness value.
22. The information processing device according to claim 17, wherein the correction unit corrects the gain information for each of the plurality of control viewpoints based on a production loudness value specified for each control viewpoint, the measured loudness value for the control viewpoint, and the target loudness value.
23. The information processing device described in claim 22, wherein the correction unit calculates a common correction amount based on the production loudness value and the target loudness value of each of the multiple control viewpoints, and corrects the gain information of the control viewpoint based on the common correction amount, the production loudness value of the control viewpoint, and the measured loudness value of the control viewpoint.
24. The information processing device according to claim 23, wherein the correction unit calculates the common correction amount based on the maximum value of the production loudness values of the plurality of control viewpoints and the target loudness value.
25. The information processing device according to claim 24, wherein the correction unit corrects the gain information so that the loudness of the audio data of the content when the control viewpoint at which the production loudness value is at the maximum value is set as the position of the listener becomes the target loudness value.
26. The information processing device according to claim 17, further comprising an acquisition unit that acquires configuration information including the measured loudness value of the control viewpoint.
27. The information processing device of claim 26, wherein the configuration information includes information for specifying whether to correct the gain information based on the measured loudness value and the target loudness value, whether to correct the gain information based on the measured loudness value, information indicating the group to which the control viewpoint belongs that is included in the configuration information, and the target loudness value, or whether to correct the gain information based on the measured loudness value, a production loudness value specified for each control viewpoint that is included in the configuration information, and the target loudness value.
28. The information processing device according to claim 26, wherein the acquisition unit acquires a plurality of pieces of configuration information, and the correction unit corrects the gain information using one piece of configuration information selected from the plurality of pieces of configuration information.
29. The information processing device of claim 28, wherein the correction unit selects the configuration information in response to an operation of the listener, or selects the configuration information based on at least one of the position of the listener in the space, the resources of the information processing device, the remaining battery power of the information processing device, the device type of the information processing device, the state of the network through which the content data is transmitted, and the scene of the content.
30. The information processing device of claim 17, wherein the correction unit calculates listener reference gain information, which is the gain information of the object relative to the position of the listener, based on the corrected gain information of the multiple control viewpoints, position information indicating the positions of the multiple control viewpoints in the space, and listener position information indicating the position of the listener in the space.
31. The information processing device according to claim 30, further comprising a rendering processing unit that generates audio data of the content based on the listener reference gain information of the object and the audio data of the object.
32. The information processing device according to claim 17, wherein the object is arranged at a different position in the space for each of the control viewpoints.
33. The information processing device according to claim 17, further comprising a control unit that displays an image of the space in which the object is arranged.
34. The information processing device according to claim 33, wherein the image displays an area for inputting the target loudness value.
35. An information processing method in which an information processing device corrects gain information of an object, determined for a control viewpoint in a space, based on a measured loudness value, which is a measurement result of the loudness of audio data of content including sounds of one or more objects when the control viewpoint is set to the position of a listener, and a predetermined target loudness value.
36. An information processing device having a control unit that generates configuration information including a measured loudness value that is a measurement result of the loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener.
37. An information processing method in which an information processing device generates configuration information including a measured loudness value that is a measurement result of the loudness of audio data of content including sounds of one or more objects when a control viewpoint in a space is set to the position of a listener.
38. An information processing device comprising: a generation unit that generates a bitstream including configuration information storing measured loudness values that are the measurement results of the loudness of audio data of content including the sounds of one or more objects when a control viewpoint in a space is set to the position of a listener; and a communication unit that transmits the bitstream.
Citation Information
Patent Citations
Apparatus and method for generating low frequency channels
JP2007512740A
A system that synthesizes loudness measurements in a single playback mode.
JP2013521539A
Loudness measuring device and loudness measurement method
JP2015158543A
Loudness Control for User Interaction in Audio Coding Systems
JP2018524630A
Optimizing loudness and dynamic range across different playback devices
JP2020095290A
