Signaling of audio effect metadata in a bitstream

Abstract

A method, system, computer-readable medium, and device for manipulating a sound field are provided. Some configurations include receiving a bitstream containing metadata and a sound field description; parsing the metadata to obtain an effect identifier and at least one effect parameter value; and applying an effect identified by the effect identifier to the sound field description. The application may include using at least one effect parameter value to apply the identified effect to the sound field description.

Description

Technology Field

[0001] This application claims priority to Greek provisional patent application No. 20190100493, titled "SIGNALLING OF AUDIO EFFECT METADATA IN A BITSTREAM," filed on [date], [date], [date], the entirety of which is incorporated herein by reference.

[0002] Aspects of the present disclosure relate to audio signal processing. Background Technology

[0003] The advancement of surround sound has made various output formats available for entertainment today. The various surround sound formats on the market include the popular 5.1 home theater system format. This format has been the most successful in expanding beyond stereo into the living room. This format includes the following six channels: Front Left (L), Front Right (R), Center or Front Center (C), Rear Left or Surround Left (Ls), Rear Right or Surround Right (Rs), and Low Frequency Effects (LFE). Other examples of surround sound formats include the 7.1 format developed by NHK (Nippon Hoso Kyokai or Japan Broadcasting Corporation) for use with Ultra High Definition television standards, and the future-oriented 22.2 format. It may be desirable for surround sound formats to encode audio in two dimensions (2D) and / or three dimensions (3D). However, these 2D and / or 3D surround sound formats require high bit rates to properly encode audio in 2D and / or 3D.

[0004] In addition to channel-based expressions, new audio formats for enhanced playback are becoming available, such as object-based and scene-based codecs (e.g., Higher-Order Ambisonics (Ambisonics) or HOA). Audio objects encapsulate individual Pulse Code Modulation (PCM) audio streams along with three-dimensional (3D) positional coordinates and other spatial information (e.g., object coherence) encoded as metadata. PCM streams are typically encoded using transform-based methods (e.g., MPEG Layer-3 (MP3), AAC, MDCT-based coding). Metadata may also be encoded for transmission. At the decoding and rendering stages, the metadata is combined with the PCM data to recreate a 3D sound field.

[0005] Scene-based audio is typically encoded using ambisonic formats such as B format. The channels of a B format signal correspond to square harmonic basis functions of the sound field, rather than loudspeaker feeds. A first-order B format signal has up to four channels (an omnidirectional channel W and directional channels X, Y, Z); a second-order B format signal has up to nine channels (four first-order channels and five additional channels R, S, T, U, V); a third-order B format signal has up to sixteen channels (nine second-order channels and seven additional channels K, L, M, N, O, P, Q).

[0006] Advanced audio codecs (e.g., object-based codecs or scene-based codecs) can be used to represent the sound field (i.e., the distribution of air pressure in space and time) across a region to support multi-directional and immersive playback. The integration of a head-related transfer function (HRTF) during rendering can be used to enhance these qualities of headphones. means of solving the problem

[0007] A method for manipulating a sound field according to a general configuration comprises: receiving a bitstream including metadata and a sound field description; parsing the metadata to obtain an effect identifier and at least one effect parameter value; and applying an effect identified by the effect identifier to the sound field description. The step of applying may include using at least one effect parameter value to apply the identified effect to the sound field description. A computer-readable storage medium is also disclosed that includes code that causes the at least one processor to perform such a method when executed by at least one processor.

[0008] An apparatus for manipulating a sound field according to a general configuration includes: a decoder configured to receive a bitstream including metadata and a sound field description and to parse the metadata to obtain an effect identifier and at least one effect parameter value; and a renderer configured to apply an effect identified by the effect identifier to the sound field description. The renderer may be configured to use at least one effect parameter value to apply the identified effect to the sound field description. An apparatus is also disclosed comprising a memory configured to store computer-executable instructions and a processor coupled to the memory and configured to perform such parsing and rendering operations by executing computer-executable instructions. Brief explanation of the drawing

[0009] Aspects of the present disclosure are illustrated by way of example. In the accompanying drawings, similar reference numerals indicate similar elements. Figure 1 illustrates an example of user instructions for manipulating a sound field. Figure 2a illustrates the sequence of audio content creation and playback. FIG. 2b illustrates the sequence of audio content creation and playback according to a general configuration. FIG. 3a illustrates a flowchart of a method (M100) according to a general configuration. Figure 3b shows an example of two metadata fields related to audio effects. Figure 3c shows an example of three metadata fields related to audio effects. Figure 3d shows an example of a value table for the effect identifier metadata field. Figure 4a illustrates an example of a sound field containing three sound sources. Figure 4b shows the result of performing a focus operation on the sound field of Figure 4a. Figure 5a shows an example of rotating the sound field with respect to the reference direction. Figure 5b shows an example of displacing the reference direction of the sound field in a different direction. Figure 6a illustrates an example of a sound field and the desired translation of the user position. Figure 6b shows the result of applying the desired translation to the sound field of Figure 6a. Figure 7a shows an example of three metadata fields related to audio effects. Figure 7b shows an example of four metadata fields related to audio effects. Figure 7c shows a block diagram of the implementation of method M100, M200. Figure 8a shows an example of a user wearing a user tracking device. FIG. 8b illustrates motion (e.g., user) in 6 degrees of freedom (6DOF). Figure 9a illustrates an example of a restriction flag metadata field associated with multiple effect identifiers. FIG. 9b illustrates an example of multiple restriction flag metadata fields associated with corresponding effect identifiers. Figure 9c illustrates an example of a restriction flag metadata field associated with a duration metadata field. Figure 9d illustrates an example of encoding audio effect metadata within an extended payload. FIG. 10 illustrates examples of different levels of zooming and / or knurling for different hotspots. Figure 11a illustrates an example of a sound field containing five sound sources surrounding a user's location. Figure 11b shows the result of performing an angle compression operation on the sound field of Figure 11a. FIG. 12a illustrates a block diagram of a system according to a general configuration. FIG. 12b illustrates a block diagram of a device (A100) according to a general configuration. FIG. 12c illustrates a block diagram of an implementation (A200) of a device (A100). FIG. 13a illustrates a block diagram of a device (F100) according to a general configuration. FIG. 13b shows a block diagram of an implementation (F200) of a device (F100). Figure 14 illustrates an example of scene space. FIG. 15 illustrates an example (400) of a VR device. FIG. 16 is a drawing showing an example of an implementation (800) of a wearable device. FIG. 17 illustrates a block diagram of a system (900) that can be implemented within a device. Specific details for implementing the invention

[0010] A sound field as described in this specification may be two-dimensional (2D) or three-dimensional (3D). One or more arrays used to capture the sound field may include a linear array of transducers. Additionally or alternatively, one or more arrays may include a spherical array of transducers. One or more arrays may also be located within scene space, and such arrays may include arrays having a fixed position and / or arrays having a position that may change during an event (e.g., mounted on a person, wire, or drone). For example, one or more arrays within scene space may be mounted on people participating in the event, such as athletes and / or officials (e.g., referees) of a sports event, or musicians and / or orchestra conductors of a music event.

[0011] A sound field can be recorded using multiple distributed arrays of transducers (e.g., microphones) to capture spatial audio across a wide scene space (e.g., a baseball stadium, soccer field, or cricket stadium illustrated in the diagram). For example, capture can be performed using one or more arrays of sound detection transducers (e.g., microphones) located outside the scene space (e.g., along the periphery of the scene space). The arrays can be positioned (e.g., oriented and / or distributed) so that specific areas of the sound field are sampled more or less densely than other areas (e.g., depending on the importance of the area of ​​interest). This positioning can be changed over time (e.g., to respond to changes in the focus of interest). The arrays can vary depending on the size / type of the field or to achieve maximum coverage while reducing blind spots. The generated sound field may include audio captured from other sources (e.g., a commentator in a broadcast booth) and added to the sound field of the scene space.

[0012] Audio formats that provide more accurate modeling of the sound field (e.g., object-based and scene-based codecs) may also allow for spatial manipulation of the sound field. For example, a user may prefer to modify the played sound field in one or more of the following modes: making sounds arriving from a specific direction louder or quieter compared to sounds arriving from other directions; hearing sounds arriving from a specific direction more clearly compared to sounds arriving from other directions; hearing sounds from only one direction and / or muting sounds from a specific direction; rotating the sound field; moving the source within the sound field; moving the user's position within the sound field. The user selections or modifications described herein may be performed using, for example, a mobile device (e.g., a smartphone), a tablet, or any other interactive device or device.

[0013] Such user interactions or instructions (e.g., sound field rotation, zooming in on an audio scene) may be performed in a manner similar to selecting a region of interest in an image or video (e.g., as illustrated in the figure). The user may indicate desired audio operations on a touchscreen by performing, for example, a pull-and-hold gesture ("reverse pinch" or "open pinch") or a touch-and-hold gesture to indicate a desired zoom, or a touch-and-drag gesture to indicate a desired rotation. The user may indicate desired audio operations by hand gestures (e.g., for optical and / or sound wave detection) by moving a finger or hand away in a desired direction to indicate a zoom, or by performing a grab-and-move gesture to indicate a desired rotation. The user may indicate desired audio operations by changing the position and / or orientation of a handheld device capable of recording such changes, such as a smartphone or other device equipped with an Inertial Measurement Unit (IMU) (e.g., including one or more accelerometers, gyroscopes, and / or magnetometers).

[0014] Although audio manipulation (e.g., zooming, focusing) has been described above as a process for the consumer, it may be desirable for content creators to apply these effects during the production of media content that includes sound fields. Examples of such content may include recordings of live events, such as sports or music performances, and recordings of scripted events, such as movies or plays. The content may be audiovisual (e.g., video or film) or audio-only (e.g., recording of a music concert) and may include one or both of recorded (i.e., captured) audio and generated (e.g., synthesized, meaning synthesized rather than captured) audio. Content creators may wish to manipulate the recorded and / or generated sound fields for various reasons, such as for dramatic effects, to provide emphasis, to draw the listener's attention, to improve clarity, and so on. The product of such processing is audio content (e.g., a file or bitstream) with the intended audio effects baked in (as illustrated in FIG. 2a).

[0015] While generating audio content in this manner allows the sound field to be reproduced as intended by the content creator, such generation may also hinder the user from experiencing other aspects of the originally recorded sound field. For example, the result of a user's attempt to expand the area of ​​the sound field may not be optimal because the audio information for that area may no longer be available within the generated content. Generating audio content in this way can prevent consumers from reversing the creator's manipulations and even prevent the content creator from modifying the generated content as desired. For instance, a content creator may not be satisfied with the audio manipulation and may wish to change the effects later. Since the audio information necessary to support such changes may have been lost during generation, it may be necessary for the original sound field to be saved separately as a backup to allow for changes after creation (for example, the creator may need to maintain a separate archive of the sound field before the effects are applied).

[0016] The systems, methods, apparatuses, and devices disclosed herein may be implemented to signal intended audio manipulation as metadata. For example, captured audio content may be stored in a raw format (i.e., without intended audio effects), and the creator's intended audio effect behavior may be stored as metadata in the bitstream. A consumer of the content may decide whether to listen to the raw audio (as illustrated in FIG. 2b) or to listen to audio with the creator's intended audio effects. If the consumer selects the creator's audio effect version, audio rendering will process the audio based on the signaled audio effect behavior metadata. If the consumer selects the raw version, the consumer may also be allowed to freely apply audio effects to the raw audio stream.

[0017] Now, several exemplary configurations will be described in connection with the accompanying drawings, which form part of this specification. Although specific configurations in which one or more aspects of the present disclosure may be implemented are described below, other configurations may be used and various modifications may be made without departing from the scope of the present disclosure or the spirit of the appended claims.

[0018] Unless explicitly limited by the context, the term “signal” is used herein to denote any of its general meanings, including the state of a memory location (or set of memory locations) as expressed on a wire, bus, or other transmission medium. Unless explicitly limited by the context, the term “to occur” is used herein to denote any of its general meanings, such as to calculate or otherwise generate. Unless explicitly limited by the context, the term “to calculate” is used herein to denote any of its general meanings, such as to calculate, evaluate, estimate, and / or select from a plurality of values. Unless explicitly limited by the context, the term “to obtain” is used to denote any of its general meanings, such as to calculate, derive, receive (e.g., from an external device), and / or retrieve (e.g., from an array of storage elements). Unless explicitly limited by the context, the term “selecting” is used to denote any of its general meanings, such as identifying, indicating, applying, and / or using, less than at least one and all of the set above. Unless explicitly limited by the context, the term “determining” is used to denote any of its general meanings, such as judging, establishing, concluding, calculating, selecting, and / or evaluating. Where the term “comprising” is used in this Description and claims, it does not exclude other elements or actions.(As in “A is based on B”) The term “based on” is used to denote any of its general meanings, including cases where it is (i) “derived from” (e.g., “B is a precursor of A”), (ii) “at least based on” (e.g., “A is based on at least B”), and, where appropriate in a particular context, (iii) “identical to” (e.g., “A is identical to B”). Similarly, the term “in response to” is used to denote any of its general meanings, including “at least in response to”. Unless otherwise specified, “at least one of A, B, and C,” “one or more of A, B, and C,” “at least one of A, B, and C,” and “one or more of A, B, and C” denote “A and / or B and / or C”. "Each of A, B, and C" and "each of A, B, and C" represent "A, B, and C" unless otherwise specified.

[0019] Unless otherwise indicated, any initiation of the operation of a device having a particular feature is also explicitly intended to disclose a method having a similar feature (and vice versa), and any initiation of the operation of a device according to a particular configuration is also explicitly intended to disclose a method according to a similar configuration (and vice versa). The term “configuration” may be used to refer to a method, device, and / or system as indicated by the particular context. The terms “method,” “process,” “procedure,” and “technique” are used generally and interchangeably unless otherwise indicated by the particular context. A “task” having multiple subtasks is also a method. The terms “device” and “device” are also used generally and interchangeably unless otherwise indicated by the particular context. The terms “element” and “module” are typically used to indicate a part of a larger configuration. Unless explicitly limited by the context, the term “system” is used herein to indicate any of its ordinary meanings, including “a group of elements interacting to achieve a common purpose.”

[0020] Unless initially introduced by a definite article, ordinal terms used to modify a claim element (e.g., "first," "third," etc.) do not indicate any priority or order of that claim element with respect to other elements alone, but rather merely distinguish that claim element from other claim elements having the same name (in the absence of the use of ordinal terms). Unless explicitly limited by the context, the terms "plural" and "set" are used herein to denote integer quantities greater than 1.

[0021] FIG. 3a illustrates a flowchart of a method (M100) for manipulating a sound field according to a general configuration including tasks T100, T200, and T300. Task T100 receives a bitstream containing metadata (e.g., one or more metadata streams) and a sound field description (e.g., one or more audio streams). For example, the bitstream may include separate audio and metadata streams formatted to comply with ITU-R (International Telecommunications Union Recommendation) BS 2076-1 (Audio Definition Model, June 2017).

[0022] A sound field description may include different audio streams for different regions based on predetermined regions of interest within the sound field (e.g., an object-based approach for some regions and an HOA approach for others). For example, it may be desirable to encode regions with high wavefield intensity using an object-based or HOA approach, and to encode regions with low wavefield intensity (e.g., ambient, crowd noise, applause) using HOA or plane wave expansion.

[0023] Object-based methods can reduce sound sources to point sources, and directional patterns (e.g., variations in the direction of sound emitted by a shouter or trumpeter) may not be preserved. HOA methods (more generally, encoding methods based on a hierarchical set of basis function coefficients) are generally more efficient at encoding a larger number of sound sources than object-based methods (e.g., more objects can be represented by smaller HOA coefficients compared to object-based methods). Benefits of using HOA methods may include the ability to evaluate and / or represent sound fields at different listener positions without the need to detect and track individual objects. Rendering of HOA-encoded audio streams is generally flexible and agnostic regarding loudspeaker configurations. HOA encoding is also generally valid under free-field conditions, allowing the transformation of the user's virtual listening position to be performed within a valid area close to the nearest source.

[0024] Task T200 parses metadata to obtain an effect identifier and at least one effect parameter value. Task T300 applies the effect identified by the effect identifier to the sound field description. Information signaled in the metadata stream may include one or more types of audio effects to be applied to the sound field, e.g., focus, zoom, null rotation, and translation. For each effect to be applied, the metadata may be implemented to include a corresponding effect identifier ID10 that identifies the effect (e.g., different values ​​for each of zoom, null, focus, rotation, and translation; a mode indicator indicating a desired mode, such as a conference or conference mode, etc.). Alternatively, a unique identifier value may be assigned to each of a number of different audio effects, and also one or more special configurations or modes (e.g., the conference or meeting mode described below; transition modes, such as fade-in or fade-out; modes for mixing out one or more sound sources and / or mixing in one or more additional sound sources; Shows an example of a table of values ​​for effect identifier ID10 that provides signaling (such as a mode that enables or disables reverberation and / or equalization).

[0025] For each identified effect, metadata may include a corresponding set of effect parameter values ​​(PM10) for parameters that define how the identified effect should be applied (e.g., as illustrated in FIG. 3b). These parameters may include, for example, an indication of a region of interest for the associated audio effect (e.g., the spatial orientation and size and / or width of the region); one or more values ​​for effect-specific parameters (e.g., the intensity of the focus effect); etc. Examples of these parameters are discussed in more detail below with reference to specific effects.

[0026] It may be desirable to allocate more bits of the metadata stream to convey parameter values ​​for a single effect than for other effects. In one example, the number of bits allocated to parameter values ​​for each effect is a fixed value of the encoding scheme. In another example, the number of bits allocated to parameter values ​​for each identified effect is indicated within the metadata stream (e.g., as illustrated in the figure).

[0027] The focus effect can be defined as enhanced directivity of a specific source or region. Parameters defining how the desired focus effect is applied may include the orientation of the focus region or source, the intensity of the focus effect, and / or the width of the focus region. The orientation may be expressed in three dimensions, for example, as an azimuth and an elevation angle corresponding to the center of the region or source. In one example, the focus effect is applied during rendering by decoding the source or region of the focus at a higher HOA order (more generally, by adding one or more levels of a hierarchical set of basis function coefficients) and / or by decoding another source or region at a lower HOA order. Fig. 4a illustrates an example of a sound field to which focus is applied for source SS10, and Fig. 4b shows an example of the same sound field after the focus effect has been applied (note that the sound sources shown in the sound field diagrams herein may represent, for example, audio objects in object-based representations or virtual sources in scene-based representations). In this example, a focusing effect is applied by increasing the directivity of the source (SS10) and increasing the diffusivity of other sources (SS20, SS30).

[0028] By applying a zoom effect, the acoustic level of the sound field can be boosted in a desired direction. Parameters defining how the desired zoom effect is applied may include the direction of the area to be boosted. This direction may be represented in three dimensions, for example, as an azimuth and an elevation angle corresponding to the center of the area. Other parameters defining the zoom effect that may be included in the metadata may include one or both of the intensity of the level boost and the size of the area to be boosted (e.g., width). For a zoom effect implemented using a beamformer, the defining parameters may include the selection of the beamformer type (e.g., FIR or IIR); the selection of a set of beamformer weights (e.g., a series of one or more tap weights); a time-frequency masking value; and the like.

[0029] By applying a null effect, the acoustic level of the sound field can be reduced in the desired direction. The parameter defining how to apply the desired null effect may be similar to the parameter defining how to apply the desired zoom effect.

[0030] A rotation effect can be applied by rotating the sound field to a desired orientation. The parameter defining the desired rotation of the sound field may indicate the direction in which it should be rotated to a defined reference direction (as illustrated in FIG. 5a). Alternatively, the desired rotation may be represented as a rotation of the reference direction to a different specific direction within the sound field (as equivalently illustrated in FIG. 5b, for example).

[0031] A transformation effect may be applied to transform a sound source to a new location within the sound field. Parameters defining the desired transformation may include direction and distance (or alternatively, a rotation angle relative to the user position). FIG. 6a illustrates an example of a sound field with three sound sources SS10, SS20, SS30 and a desired transformation TR10 of source SS20; FIG. 6b illustrates the sound field after the transformation TR10 has been applied.

[0032] Each sound field change indicated in the metadata may be linked to a specific moment in the sound field stream (e.g., by a timestamp included in the metadata, as illustrated in FIGS. 7a and 7b). In the case of an implementation where more than one sound field change is indicated under a shared timestamp, the metadata may also include information to identify the time precedence between the changes (e.g., “applying the indicated rotation effect to the sound field, then applying the indicated focus effect to the rotated sound field”).

[0033] As mentioned above, it may be desirable to allow the user to select a raw version of the sound field or a version modified by audio effect metadata, and / or to modify the sound field in a manner partially or completely different from the effect indicated in the effect metadata. The user may actively provide such commands, for example, on a touchscreen, by gesture, by voice commands, etc. Alternatively or additionally, user commands may be generated by passive user interaction through a user tracking device that tracks the user's movement and / or orientation, for example, a device that may include an inertial measurement unit (IMU). FIG. 8a illustrates UT10, an example of such a device that also includes a display screen and headphones. The IMU may include one or more accelerometers, gyroscopes, and / or magnetometers to indicate and quantify movement and / or orientation.

[0034] FIG. 7c illustrates a flowchart of an implementation M200 of method M100, comprising an implementation T350 of task T400 and task T300. Task T400 receives at least one user command (e.g., by active and / or passive user interaction). Based on at least one of (A) at least one effect parameter value or (B) at least one user command, task T350 applies an effect identified by an effect identifier to a sound field description. Method M200 may be performed, for example, by an implementation of a user tracking device UT10 that receives audio and metadata streams and generates audio corresponding to the user through headphones.

[0035] To support an immersive VR experience, it may be desirable to adjust the provided audio environment in response to changes in the listener's virtual position. For example, it may be desirable to support virtual movement in 6 degrees of freedom (6DOF). As shown in FIGS. 8a and 8b, 6DOF includes three rotational movements of 3DOF and also three translational movements, namely forward / backward (surge), up / down (heave), and left / right (sway). Examples of 6DOF applications include virtual attendance at a crowd event, such as a sports event (e.g., a baseball game), by a remote user. For a user wearing a device such as the user tracking device UT10, it may be desirable to perform sound field rotation according to manual user commands generated by the device UT10 (e.g., indicating the user's current forward direction as the desired reference direction for the sound field) rather than following rotation effects displayed by the content creator in the metadata stream as described above.

[0036] It may be desirable to allow content creators to limit the extent to which effects described in metadata can be modified downstream. For example, it may be desirable to impose spatial restrictions to allow users to apply effects only in specific areas and / or to prevent users from applying effects in specific areas. These restrictions may apply to all signaled effects or a specific set of effects, or the restrictions may apply to a single effect. In one example, a spatial restriction allows users to apply a zoom effect only in a specific area. In another example, a spatial restriction prevents users from applying a zoom effect in other specific areas (e.g., confidential and / or private areas). In yet another example, it may be desirable to impose spatial restrictions to allow users to apply effects only during specific intervals and / or to prevent users from applying effects during specific intervals. Again, these restrictions may apply to all signaled effects or a specific set of effects, or the restrictions may apply to a single effect.

[0037] To support these restrictions, metadata may include flags indicating the desired restrictions. For example, a restriction flag may indicate whether one or more (if possible, all) of the effects indicated in the metadata can be overwritten by user interaction. Additionally or alternatively, a restriction flag may indicate whether user modification of the sound field is allowed or disabled. Such disabling may apply to all effects, or one or more effects may be specifically enabled or disabled. Restrictions may apply to the entire file or bitstream, or may be associated with a specific period within the file or bitstream. In another example, effect identifiers may be implemented to use different values ​​to distinguish between a restricted version of an effect (e.g., which cannot be removed or overwritten) and an unrestricted version of the same effect (which may be applied or ignored at the consumer's discretion).

[0038] FIG. 9a illustrates an example of a metadata stream in which a restriction flag (RF10) is applied to two identified effects. FIG. 9b illustrates an example of a metadata stream in which a separate restriction flag is applied to each of two different effects. FIG. 9c illustrates an example in which a restriction flag is accompanied by a restriction duration (RD10) indicating the duration of time during which the restriction is valid.

[0039] An audio file or stream may contain one or more versions of effect metadata, and different versions of this effect metadata may be provided for the same audio content (e.g., as user suggestions from a content creator). Different versions of effect metadata may provide different focus areas for different audiences, for example. In one example, different versions of effect metadata may describe the effect of zooming in on different people (e.g., actors, athletes) in a video. A content creator may mark up interesting audio sources and / or orientations (e.g., different levels of zooming and / or knurling for different hotspots, as illustrated in FIG. 10), and the corresponding video stream may be configured to support user selection of the desired metadata stream by selecting corresponding features from the video stream. In another example, different versions of user-generated metadata may be shared via social media (e.g., for live events with many different audience perspectives, such as stadium-scale music events). Different versions of effect metadata may describe different variations of the same sound field corresponding to different video streams, for example. Different versions of the audio effect metadata bitstream can be downloaded or streamed separately from a source different from the sound field itself.

[0040] Effect metadata may be generated automatically based on human instructions (e.g., by content creators) and / or one or more design criteria. For example, in a remote conferencing application, it may be desirable to automatically select the largest single audio source or audio from multiple story sources and to lessen the emphasis on other audio components of the sound field (e.g., discard or lower the volume). The corresponding effect metadata stream may include a flag indicating a "conference mode." In an example such as illustrated in FIG. 3c, one or more of the possible values ​​of the effect identifier field of the metadata (e.g., effect identifier ID10) are assigned to indicate the selection of such a mode. Parameters defining how the conference mode is applied may include the number of sources to zoom in on (e.g., the number of people at the conference table, the number of people to speak, etc.). The number of sources may be selected by the on-site user, by the content creator, and / or automatically. For example, face, motion, and / or person detection may be performed on one or more corresponding video streams to identify the direction of interest and / or to support the suppression of noise arriving from other directions.

[0041] Other parameters defining how the conference mode should be applied may include metadata (e.g., beamformer weights, time-frequency masking values, etc.) to enhance source extraction from the sound field. The metadata may also include one or more parameter values ​​indicating the desired rotation of the sound field. The sound field may be rotated according to the position of the largest audio source to support automatic rotation of the remote user's video and audio so that, for example, the largest speaker is in front of the remote user. In another example, the metadata may indicate automatic rotation of the sound field so that a two-person discussion takes place in front of the remote user. In additional examples, the parameter values ​​may indicate compression (or other remapping) of the recorded angular range of the sound field (e.g., as illustrated in FIG. 11a) so that the remote participant can perceive the other attendee as being in front of her rather than behind her (as illustrated in FIG. 11b).

[0042] The audio effect metadata stream described herein may be carried in the same transmission as the corresponding audio stream (or streams), in a separate transmission, or even received from a different source (e.g., as described above). In one example, the effect metadata stream is stored or transmitted in a dedicated extension payload (e.g., in the afx_data field shown in FIG. 9d), which is an existing feature of the Advanced Audio Coding (AAC) codec (e.g., defined in ISO / IEC 14496-3:2009) and more recent codecs. The data in this extension payload may be processed by a device that understands this type of extension payload (e.g., a decoder and a renderer) and may be ignored by other devices. In another example, the audio effect metadata stream described herein may be standardized for an audio or audiovisual codec. This approach may be implemented, for example, as a correction in audio groups as part of a standardized representation of immersive environments (e.g., MPEG-H (as described, e.g., in ATSC (Advanced Television Systems Committee) Doc. A / 342-3:2017) and / or MPEG-I (e.g., as described in ISO / IEC 23090). In additional examples, the audio effect metadata streams described herein may be implemented according to the CICP (coding-independent code point) specification. Additional use cases for the audio effect metadata streams described herein include encoding within the IVAS (Immersive Voice and Audio Services) codec (e.g., as part of a 3GPP implementation).

[0043] Although described in relation to AAC, the techniques may be performed using any type of psychoacoustic audio coding that allows for extended payloads and / or extended packets (e.g., fill elements of information containing identifiers followed by fill data or other containers) or otherwise allows backward compatibility, as described in more detail below. Examples of other psychoacoustic audio codecs include Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 ALS (Audio Lossless Streaming), aptX® Enhanced AC-3, FLAC (Free Lossless Audio Codec), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and WMA (Windows Media Audio).

[0044] FIG. 12a illustrates a block diagram of a system for processing a bitstream containing audio data and audio effect metadata described herein. The system includes an audio decoding stage configured to parse audio effect metadata (e.g., received in an extended payload) and provide the metadata to an audio rendering stage. The audio rendering stage is configured to apply audio effects as intended by the creator using the audio effect metadata. The audio rendering stage may also be configured to receive user interactions to manipulate audio effects and (if permitted) consider these user commands.

[0045] FIG. 12b illustrates a block diagram of a device (A100) according to a general configuration including a decoder (DC10) and a sound field renderer (SR10). The decoder DC10 is configured to receive a bitstream BS10 containing metadata MD10 and a sound field description SD10 (e.g., as described herein in relation to task T100) and to parse the metadata MD10 to obtain an effect identifier and at least one effect parameter value (e.g., as described herein in relation to task T200). The renderer SR10 is configured to apply an effect identified by the effect identifier to the sound field description SD10 (e.g., as described herein in relation to task T300) to generate a modified sound field MS10. For example, the renderer SR10 may be configured to use at least one effect parameter value to apply the identified effect to the sound field description SD10.

[0046] Renderer SR10 may be configured to apply a focus effect to a sound field, for example, by rendering a selected area of ​​the sound field at a higher resolution than other areas and / or by rendering other areas to have higher diffusion. In one example, a device or machine performing task T300 (e.g., renderer SR10) is configured to implement the focus effect by requesting additional information (e.g., higher-order HOA coefficient values) about the focus source or area from a server via a wired connection and / or a wireless connection (e.g., Wi-Fi and / or LTE).

[0047] Renderer SR10 may be configured to apply a zoom effect to the sound field by applying a beamformer, for example (e.g., according to parameter values ​​returned within the corresponding fields of metadata). Renderer SR10 may be configured to apply a rotation or transformation effect to the sound field by applying a matrix transformation corresponding to a set of HOA coefficients (or more generally, a hierarchical set of basis function coefficients) and / or thereby moving audio objects within the sound field.

[0048] FIG. 12c illustrates a block diagram of an implementation (A200) of a device (A100) comprising a command processor CP10. The processor CP10 is configured to receive metadata MD10 and at least one user command UC10 as described herein, and to generate at least one effect command EC10 based on at least one effect parameter value (e.g., according to one or more restriction flags in the metadata). A renderer SR10 is configured to use at least one effect command EC10 to apply an effect identified in a sound field description SD10 to generate a modified sound field MS10.

[0049] FIG. 13a illustrates a block diagram of a device (F100) for manipulating a sound field according to a general configuration. The device (F100) includes means MF100 for receiving a bitstream comprising metadata (e.g., one or more metadata streams) and a sound field description (e.g., one or more audio streams) (e.g., as described herein in relation to task T100). For example, the receiving means (MF100) includes a transceiver, a modem, a decoder (DC10), one or more other circuits or devices configured to receive the bitstream (BS10), or a combination thereof. The device F100 also includes means MF200 for parsing the metadata to obtain an effect identifier and at least one effect parameter value (e.g., as described herein in relation to task T200). For example, the parsing means (MF200) includes a decoder (DC10), one or more other circuits or devices configured to parse the metadata (MD10), or a combination thereof. Device F100 also includes means MF300 for applying an effect identified by an effect identifier to a sound field description (e.g., as described herein in relation to task T300). For example, means MF300 may be configured to apply the identified effect by using at least one effect parameter value to apply a matrix transformation to the sound field description. In some examples, the means for applying the effect (MF300) includes a renderer (SR10), a processor (CP10), one or more other circuits or devices configured to apply the effect to the sound field description (SD10), or a combination thereof.

[0050] FIG. 13b illustrates a block diagram of an implementation F200 of device F100 comprising means MF400 for receiving at least one user command (e.g., by active and / or passive user interaction) (e.g., as described herein in relation to task T400). For example, means MF400 for receiving at least one user command comprises a processor CP10, one or more other circuits or devices configured to receive at least one user command UC10, or a combination thereof. Device F200 also comprises means MF350 (an implementation of means MF300) for applying an effect identified by an effect identifier to a sound field description based on at least one of (A) at least one effect parameter value or (B) at least one user command. In one example, means MF350 comprises means for combining at least one effect parameter value with a user command to obtain at least one revised parameter. In another example, parsing metadata includes parsing metadata to obtain a second effect identifier, and means MF350 includes means for determining not to apply the effect identified by the second effect identifier to the sound field description. In some examples, means for applying the effect (MF350) include a renderer (SR10), a processor (CP10), one or more other circuits or devices configured to apply the effect to the sound field description (SD10), or a combination thereof. Device F200 may be embodied, for example, by an implementation of a user tracking device UT10 that receives audio and metadata streams and generates audio corresponding to the user through headphones.

[0051] Hardware for virtual reality (VR) may include one or more screens to provide a visual scene to the user, one or more sound-emitting transducers to provide a corresponding audio environment (e.g., an array of loudspeakers or an array of head-mounted transducers), and one or more sensors to determine the user's position, orientation, and / or movement. A user tracking device UT10, as illustrated in FIG. 8a, is an example of a headset. To support an immersive experience, such a headset can detect the orientation of the user's head in three degrees of freedom (3DOF)—rotation of the head around the up-down axis (yaw), tilt of the head in the front-back plane (pitch), and tilt of the head in the left-right plane (roll)—and adjust the provided audio environment accordingly.

[0052] Computer-mediated reality systems are being developed to allow computing devices to augment or add, remove or subtract, replace or substitute, or generally alter the existing reality experienced by the user. Examples of computer-mediated reality systems may include virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of computer-mediated reality systems generally relates to the ability of such systems to provide a realistically immersive experience in terms of both video and audio, ensuring that the video and audio experiences align with how they are perceived by the user in a natural and expected manner. Although the human visual system is more sensitive than the human auditory system (for example, in terms of locating the perceived position of various objects within a scene), ensuring an appropriate auditory experience is becoming an increasingly important factor in guaranteeing a realistic immersive experience, particularly as video experiences are enhanced to allow for better localization of video objects, enabling users to better identify the source of audio content.

[0053] In VR technologies, virtual information may be presented to the user using a head-mounted display so that the user can visually experience an artificial world on a screen in front of their eyes. In AR technologies, the real world is augmented by visual objects that can be superimposed (e.g., overlaid) on physical objects in the real world. Augmentation may insert new visual objects into the real-world environment or mask visual objects. In MR technologies, the boundary between what is real or synthetic / virtual and what is visually experienced by the user is becoming difficult to distinguish. The technique described herein can be used with the VR device (400) illustrated in FIG. 15 to improve the experience of the device user (402) through the device's headphones (404).

[0054] Video, audio, and other sensory data can play an important role in a VR experience. To participate in a VR experience, a user (402) may wear a VR device (400) or other wearable electronic device (which may also be referred to as a VR headset (400)). A VR client device (e.g., a VR headset (400)) can track the head movements of the user (402) and adapt video data shown through the VR headset (400) to describe the head movements, thereby providing an immersive experience in which the user (402) can experience a virtual world displayed in the video data in three visual dimensions.

[0055] While VR (and other forms of AR and / or MR) can allow a user (402) to visually inhabit a virtual world, the VR headset (400) often lacks the ability to audibly place the user in the virtual world. That is, the VR system (which may include a computer responsible for rendering video and audio data (not shown in the example of FIG. 15 for convenience of explanation) and a headset (400)) may not support full three-dimensional immersion audibly (and in some cases realistically in a way that reflects the virtual scene displayed to the user through the VR headset (400)).

[0056] While full three-dimensional audible rendering still presents challenges, the techniques in this disclosure enable further steps toward that goal. Audio modalities of AR, MR, and / or VR may be classified into three separate categories of immersion. The first category provides the lowest level of immersion and is referred to as 3 Degrees of Freedom (3DOF). 3DOF refers to audio rendering that describes head movements in degrees of freedom (yaw, pitch, and roll), thereby allowing the user to freely look around in any direction. However, 3DOF cannot describe translational (and directional) head movements in which the head is not centered at the optical and acoustic center of the sound field.

[0057] A second category, referred to as 3DOF Plus (or "3DOF+"), provides three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational (and directional) movements resulting from head movements away from the optical and acoustic centers within the sound field. 3DOF+ may also provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0058] The third category, referred to as 6 degrees of freedom (6DOF), renders audio data in a way that describes not only the 3 degrees of freedom (yaw, pitch, and roll) in terms of head movements but also the translation of a person in space (x, y, and z translations). Spatial transformation can be induced, for example, by a sensor tracking a person's position in the physical world, through an input controller, and / or through a rendering program that simulates the user's movement within virtual space.

[0059] Audio modes in VR may be less immersive than video modes, which could potentially reduce the overall immersion experienced by the user. However, with advancements in processors and wireless connectivity, it may be possible to achieve 6DOF rendering with wearable AR, MR, and / or VR devices. Furthermore, in the future, it may be possible to consider vehicle movements that provide immersive audio experiences utilizing the capabilities of AR, MR, and / or VR devices. Additionally, those skilled in the art will recognize that mobile devices (e.g., handsets, smartphones, tablets) can also implement VR, AR, and / or MR techniques.

[0060] According to the techniques described in this disclosure, various methods of adjusting audio data (whether in audio channel formats, audio object formats, and / or audio scene-based formats) may allow for audio rendering. 6DOF rendering provides a more immersive listening experience by rendering audio data in a manner that describes three degrees of freedom in terms of head movements (yaw, pitch, and roll) and also translational movements (e.g., in a spatial 3D coordinate system - x, y, z). In implementation, adjustments may be made to provide 6DOF rendering where head movements may not be centered at the optical and acoustic centers, and this is not necessarily limited to spatial 2D coordinate systems. As disclosed herein, the following drawings and descriptions allow for 6DOF audio rendering.

[0061] FIG. 16 is a drawing illustrating an example of an implementation (800) of a wearable device that may operate according to various aspects of the techniques described in the present disclosure. In various examples, the wearable device (800) may represent a VR headset (e.g., the VR headset (400) described above), an AR headset, an MR headset, or an Extended Reality (XR) headset. Augmented Reality “AR” may refer to computer-rendered images or data overlaid on the real world where the user is actually located. Mixed Reality “MR” may refer to computer-rendered images or data that are a world fixed at a specific location in the real world, or may refer to a variation of VR in which partially computer-rendered 3D elements and partially captured real elements are combined into an immersive experience that simulates the user’s physical presence in the environment. Extended Reality “XR” may refer to a comprehensive term for VR, AR, and MR.

[0062] The wearable device (800) may represent other types of devices such as watches (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, etc. Regardless of whether it represents a VR device, a watch, glasses and / or headphones, the wearable device (800) may communicate with a computing device that supports the wearable device (800) via a wired connection or a wireless connection.

[0063] In some cases, a computing device supporting the wearable device (800) may be integrated within the wearable device (800), and thus the wearable device (800) may be considered the same device as the computing device supporting the wearable device (800). In other examples, the wearable device (800) may communicate with a separate computing device capable of supporting the wearable device (800). In this regard, the term “supporting” should not be understood as requiring a separate dedicated device, and it should be understood that one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within the wearable device (800) or integrated within a computing device separate from the wearable device (800).

[0064] For example, when the wearable device (800) represents a VR device (400), a separate dedicated computing device (such as a personal computer including one or more processors) can render audio and visual content, whereas the wearable device (800) can determine the translational head movement, which allows the dedicated computing device to render audio content (as a speaker feed) according to various aspects of the technique described in this disclosure based on the translational head movement. As another example, when the wearable device (800) represents smart glasses, the wearable device (800) may include a processor (e.g., one or more processors) that determines the translational head movement (by interfacing within one or more sensors of the wearable device (800)) and renders a loudspeaker feed based on the determined translational head movement.

[0065] As described, the wearable device (800) includes a rear camera, one or more directional speakers, one or more tracking and / or recording cameras, and one or more light-emitting diode (LED) lights. In some examples, the LED light(s) may be referred to as "ultraluminal" LED light(s). Additionally, the wearable device (800) includes one or more eye-tracking cameras, high-sensitivity audio microphones, and optical / projection hardware. The optical / projection hardware of the wearable device (800) may include durable translucent display technology and hardware.

[0066] The wearable device (800) also includes connectivity hardware that may represent one or more network interfaces supporting multimode connectivity, such as 4G communication, 5G communication, etc. The wearable device (800) also includes an ambient light sensor and a bone conduction transducer. In some cases, the wearable device (800) may also include one or more passive and / or active cameras equipped with a fisheye lens and / or a telephoto lens. The steering angle of the wearable device (800) may be used to select an audio representation of the sound field (e.g., one of a mixed-order ambisonic (MOA) representation) to be output through the directional speaker(s) (headphones (404)) of the wearable device (800) according to various techniques of the present disclosure. It will be understood that the wearable device (800) may represent various different form factors.

[0067] Although not illustrated in the example of FIG. 16, the wearable device (800) may include an orientation / transformation sensor unit, for example, a microelectromechanical system (MEMS) for sensing, or a combination of any other type of sensor capable of providing information to support head and / or body tracking. In one example, the orientation / transformation sensor unit may represent a MEMS for sensing translational motion similar to those used in cellular phones such as so-called "smartphones."

[0068] Although described in relation to specific examples of wearable devices, those skilled in the art will understand that the description related to FIGS. 15 and FIGS. 16 may apply to other examples of wearable devices. For example, other wearable devices, such as smart glasses, may include sensors for obtaining translational head movements. As another example, other wearable devices, such as smartwatches, may include sensors for obtaining translational motion. As such, the techniques described in this disclosure should not be limited to specific types of wearable devices, and any wearable device may be configured to perform the techniques described in this disclosure.

[0069] FIG. 17 illustrates a block diagram of a system (900) that may be implemented within a device (e.g., a wearable device (400 or 800)). The system (900) includes a processor (420) (e.g., one or more processors) that may be configured to perform the method M100 or M200 described herein. The system (900) also includes a memory (120) coupled to a processor (420), a sensor (110) (e.g., an ambient light sensor, orientation and / or tracking sensor of the device (800)), a visual sensor (130) (e.g., a night vision sensor, a tracking and recording camera, an eye-tracking camera, and a rear camera of the device (800)), a display device (100) (e.g., an optical / projection of the device (800)), an audio capture device (112) (e.g., a high-sensitivity microphone of the device (800)), a loudspeaker (470) (e.g., headphones (404) of the device (400), a directional speaker of the device (800)), a transceiver (480), and an antenna (490). In a particular aspect, the system (900) includes a modem in addition to or alternative to the transceiver (480). For example, a modem, a transceiver (480), or both are configured to receive a signal representing bitstream BS10 and provide bitstream BS10 to a decoder DC10.

[0070] Various elements of the implementation of the device or system disclosed herein (e.g., devices A100, A200, F100, and / or F200) may be implemented in any combination of hardware and software and / or firmware considered suitable for the intended application. For example, such elements may be manufactured as electronic and / or optical devices residing, for example, on the same chip or between two or more chips of a chipset. An example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements may be implemented as one or more such arrays. Any two or more, or even all, of these elements may be implemented within the same array or arrays. Such arrays or arrays may be implemented within one or more chips (e.g., within a chipset containing two or more chips).

[0071] A processor or other means for processing as disclosed herein may be manufactured as one or more electronic and / or optical devices residing, for example, on the same chip or between two or more chips of a chipset. An example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements may be implemented as one or more such arrays. Such arrays or arrays may be implemented within one or more chips (for example, within a chipset comprising two or more chips). Examples of such arrays include fixed or programmable arrays of logic elements such as microprocessors, embedded processors, IP cores, DSPs (digital signal processors), FPGAs (field programmable gate arrays), ASSPs (application-specific standard products), and (application-specific integrated circuits). A processor or other means for processing disclosed herein may also be implemented as one or more computers (for example, machines comprising one or more arrays programmed to execute one or more sets or sequences of instructions) or other processors. The processor described herein may be used to perform a task, such as a task related to other operations of a device or system in which the processor is embedded (e.g., a voice communication device such as a smartphone or smart speaker), or to execute another set of instructions that are not directly related to the procedure of implementing method M100 or M200 (or other methods disclosed in relation to the operations of the device or system described herein). It is also possible for a part of the method disclosed herein to be performed under the control of one or more other processors.

[0072] Each of the tasks of the methods disclosed herein (e.g., methods M100 and / or M200) may be directly implemented in hardware, as a software module executed by a processor, or as a combination of both. In a typical application of an implementation of the method as disclosed herein, an array of logic elements (e.g., logic gates) is configured to perform one, two or more, or even all of the various tasks of the method. One or more (possibly all) of the tasks may be implemented in code (e.g., one or more sets of instructions) implemented in a computer program product (e.g., one or more data storage media such as a disk, flash or other non-volatile memory card, semiconductor memory chip, etc.) that is readable and / or executable by a machine (e.g., computer) comprising an array of logic elements (e.g., a processor, a microprocessor, a microcontroller, or other finite state machine). Tasks of an implementation of the method disclosed herein may be performed by two or more such arrays or machines. In these or other implementations, the tasks may be performed within a device for wireless communication, such as a cellular phone or other device having such communication capabilities. Such a device may be configured to communicate with a circuit-switched and / or packet-switched network (e.g., using one or more protocols such as VoIP). For example, such a device may include RF circuitry configured to receive and / or transmit encoded frames.

[0073] In one or more exemplary embodiments, the operations described herein may be implemented in hardware, software, firmware, or any combination thereof. Where implemented in software, such operations may be stored on or transmitted through a computer-readable medium as one or more instructions or codes. The term “computer-readable medium” includes both computer-readable storage media and communication (e.g., transmission) media. By example, but not by limitation, a computer-readable storage medium may include semiconductor memory (which may include, without limitation, dynamic or static RAM, ROM, EEPROM, and / or flash RAM), or an array of storage elements such as ferroelectric, magnetoresistive, ovonic, polymer, or phase-change memory; a CD-ROM or other optical disc storage device; and / or a magnetic disc storage device or other magnetic storage device. Such storage media may store information in the form of instructions or data structures that can be accessed by a computer. A communication medium may include any medium that can be used to carry desired program code in the form of instructions or data structures and can be accessed by a computer, including any medium that facilitates the transmission of a computer program from one place to another. Additionally, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and / or microwave, the definition of the medium includes coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and / or microwave.As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy discs, and Blu-ray discs™ (Blu-ray Disc Association, Universal City, California), wherein disks typically reproduce data magnetically, while discs reproduce data optically using a laser. Additionally, the combinations mentioned above must be included within the scope of computer-readable media.

[0074] In one example, a non-transient computer-readable storage medium comprises code that, when executed by at least one processor, causes at least one processor to perform a method of characterizing a portion of a sound field as described herein. Further examples of such storage media comprise a medium that, when executed by at least one processor, causes at least one processor to receive a bitstream containing metadata and a sound field description (e.g., as described herein with reference to task T100); to parse the metadata to obtain an effect identifier and at least one effect parameter (e.g., as described herein with reference to task T200); and to apply an effect identified by the effect identifier to the sound field description (e.g., as described herein with reference to task T300). The step of applying may include the step of using at least one effect parameter to apply the identified effect to the sound field description.

[0075] Examples of implementation are described in the following numbered clauses.

[0076] Clause 1. A method for manipulating a sound field, the method comprises the steps of: receiving a bitstream including metadata and a sound field description; parsing the metadata to obtain an effect identifier and at least one effect parameter value; and applying an effect identified by the effect identifier to the sound field description.

[0077] Clause 2. A method for manipulating a sound field according to Clause 1, wherein the step of parsing the metadata includes the step of parsing the metadata to obtain a timestamp corresponding to the effect identifier, and the step of applying the identified effect includes the step of using the at least one effect parameter value to apply the identified effect to a part of the sound field description corresponding to the timestamp.

[0078] Clause 3. A method for manipulating a sound field, wherein the step of applying the identified effect in Clause 1 comprises the step of combining the at least one effect parameter value with a user command to obtain at least one revised parameter value.

[0079] Clause 4. A method for manipulating a sound field, wherein, in any one of Clauses 1 to 3, the step of applying the identified effect includes the step of rotating the sound field in a desired orientation.

[0080] Clause 5. A method for manipulating a sound field, wherein in any one of Clauses 1 to 3, the at least one effect parameter value includes an indicated direction, and the step of applying the identified effect includes the step of using the at least one effect parameter value to rotate the sound field in the indicated direction.

[0081] Clause 6. A method for manipulating a sound field, wherein in any one of Clauses 1 to 3, the at least one effect parameter value includes an indicated direction, and the step of applying the identified effect includes the step of using the at least one effect parameter value to increase the acoustic level of the sound field in the indicated direction compared to the acoustic level of the sound field in other directions.

[0082] Clause 7. A method for manipulating a sound field, wherein in any one of Clauses 1 to 3, the at least one effect parameter value includes an indicated direction, and the step of applying the identified effect includes the step of using the at least one effect parameter value to reduce the acoustic level of the sound field in the indicated direction compared to the acoustic level of the sound field in other directions.

[0083] Clause 8. A method for manipulating a sound field, wherein in any one of Clauses 1 to 3, the at least one effect parameter value indicates a location within the sound field, and the step of applying the identified effect includes the step of using the at least one effect parameter value to convert a sound source to the indicated location.

[0084] Clause 9. A method for manipulating a sound field, wherein in any one of Clauses 1 to 3, the at least one effect parameter value includes an indicated direction, and the step of applying the identified effect includes the step of using the at least one effect parameter value to increase the directivity of at least one of the sound source of the sound field or the region of the sound field compared to another sound source of the sound field or the region of the sound field.

[0085] Clause 10. A method for manipulating a sound field, wherein, in any one of Clauses 1 to 3, the step of applying the identified effect includes the step of applying a matrix transformation to the sound field description.

[0086] Clause 11. A method of manipulating a sound field according to Clause 10, wherein the matrix transformation comprises at least one of rotation of the sound field and translation of the sound field.

[0087] Clause 12. A method for manipulating a sound field in any one of Clauses 1 to 3, wherein the sound field description comprises a hierarchical set of basis function coefficients.

[0088] Clause 13. A method for manipulating a sound field, wherein, in any one of Clauses 1 to 3, the sound field description comprises a plurality of audio objects.

[0089] Clause 14. A method for manipulating a sound field, wherein in any one of Clauses 1 to 3, the step of parsing the metadata includes the step of parsing the metadata to obtain a second effect identifier, and the method includes the step of deciding not to apply the effect identified by the second effect identifier to the sound field description.

[0090] Clause 15. A device for manipulating a sound field, the device comprises: a decoder configured to receive a bitstream including metadata and a sound field description and to parse the metadata to obtain an effect identifier and at least one effect parameter value; and a renderer configured to apply an effect identified by the effect identifier to the sound field description.

[0091] Clause 16. In Clause 15, further comprising a modem configured to receive a signal representing a bitstream; and to provide the bitstream to a decoder.

[0092] Clause 17. A device for manipulating a sound field, the device comprises: a memory configured to store a bitstream including metadata and a sound field description; and a processor coupled to the memory and configured to parse the metadata to obtain an effect identifier and at least one effect parameter value; and to apply an effect identified by the effect identifier to the sound field description.

[0093] Clause 18. A device for manipulating a sound field, wherein, in Clause 17, the processor is configured to parse the metadata to obtain a timestamp corresponding to the effect identifier and to apply the identified effect by using the at least one effect parameter value to apply the identified effect to a portion of the sound field description corresponding to the timestamp.

[0094] Clause 19. A device for manipulating a sound field, wherein, in Clause 17, the processor is configured to combine the at least one effect parameter value with a user command to obtain at least one revised parameter.

[0095] Clause 20. A device for manipulating a sound field, wherein in any one of Clauses 17 to 19, the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the identified effect by using the at least one effect parameter value to rotate the sound field in the indicated direction.

[0096] Clause 21. A device for manipulating a sound field, wherein in any one of Clauses 17 to 19, the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the identified effect by using the at least one effect parameter value to increase the acoustic level of the sound field in the indicated direction relative to the acoustic level of the sound field in other directions.

[0097] Clause 22. A device for manipulating a sound field, wherein in any one of Clauses 17 to 19, the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the identified effect by using the at least one effect parameter value to reduce the acoustic level of the sound field in the indicated direction relative to the acoustic level of the sound field in other directions.

[0098] Clause 23. A device for manipulating a sound field, wherein, in any one of Clauses 17 to 19, the at least one effect parameter value indicates a location within the sound field, and the processor is configured to apply the identified effect by using the at least one effect parameter value to convert a sound source to the indicated location.

[0099] Clause 24. A device for manipulating a sound field, wherein in any one of Clauses 17 to 19, the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the identified effect by using the at least one effect parameter value to increase the directivity of at least one of the sound source of the sound field or the region of the sound field relative to another sound source of the sound field or the region of the sound field.

[0100] Clause 25. A device for manipulating a sound field, wherein, in any one of Clauses 17 to 19, the processor is configured to apply the identified effect by using the at least one effect parameter value to apply a matrix transformation to the sound field description.

[0101] Clause 26. A device for manipulating a sound field, wherein the matrix transformation in Clause 25 comprises at least one of rotation of the sound field and translation of the sound field.

[0102] Clause 27. A device for manipulating a sound field, wherein, in any one of Clauses 17 to 19, the sound field description comprises a hierarchical set of basis function coefficients.

[0103] Clause 28. In any one of Clauses 17 to 19, the sound field description comprises a plurality of audio objects, a device for manipulating a sound field.

[0104] Clause 29. A device for manipulating a sound field, wherein, in any one of Clauses 17 to 19, the processor is configured to parse the metadata to obtain a secondary effect identifier and to decide not to apply the effect identified by the secondary effect identifier to the sound field description.

[0105] Clause 30. A device for manipulating a sound field, wherein, in any one of Clauses 17 to 19, the device comprises a custom integrated circuit including the processor.

[0106] Clause 31. A device for manipulating a sound field, the device comprises: means for receiving a bitstream including metadata and a sound field description; means for parsing the metadata to obtain an effect identifier and at least one effect parameter value; and means for applying an effect identified by the effect identifier to the sound field description.

[0107] Clause 32. In Clause 31, at least one of the receiving means, the parsing means, or the applying means is integrated into at least one of a mobile phone, a tablet computer device, a wearable electronic device, a camera device, a virtual reality headset, an augmented reality headset, or a vehicle.

[0108] Those skilled in the art will also recognize that the various exemplary logic blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of both. The various exemplary components, blocks, configurations, modules, circuits, and steps have generally been described in terms of their functions. Whether such functions are implemented as hardware or as processor-executable instructions depends on the design constraints imposed on the overall system and the specific application. Those skilled in the art may implement the described functionality in various ways for each specific application, and such implementation determinations should not be construed as causing a departure from the scope of this disclosure.

[0109] The steps of the method or algorithm described in connection with the implementations disclosed herein may be directly implemented in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor may read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a computing device or a user terminal.

[0110] The prior description is provided to enable those skilled in the art to manufacture or use the disclosed embodiments. Various variations of these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Accordingly, this disclosure is not intended to be limited to the embodiments shown herein, but should be granted the broadest possible scope consistent with the principles and novel characteristics defined by the following claims.

Claims

Claim 1 A method for manipulating a sound field, performed by a device for manipulating a sound field, comprising: receiving a bitstream including metadata and a sound field description, wherein the metadata includes a first effect identifier specifying a type of first effect to be applied to the sound field description and at least one effect parameter value defining how to apply the first effect, and a second effect identifier specifying a type of second effect to be applied to the sound field description; parsing the metadata to obtain the first effect identifier, the second effect identifier, and the at least one effect parameter value; deciding not to apply the second effect identified by the second effect identifier to the sound field description; and applying the first effect identified by the first effect identifier to the sound field description. Claim 2 A method for manipulating a sound field according to claim 1, wherein the step of parsing the metadata includes the step of parsing the metadata to obtain a timestamp corresponding to the first effect identifier, and the step of applying the first effect includes the step of using the at least one effect parameter value to apply the first effect to the part of the sound field description corresponding to the timestamp. Claim 3 A method for manipulating a sound field according to claim 1, wherein the step of applying the first effect comprises combining the at least one effect parameter value with a user command to obtain at least one revised parameter value. Claim 4 A method for manipulating a sound field according to claim 1, wherein the step of applying the first effect includes the step of rotating the sound field in a desired orientation. Claim 5 A method for manipulating a sound field according to claim 1, wherein the at least one effect parameter value includes an indicated direction, and the step of applying the first effect includes the step of using the at least one effect parameter value to rotate the sound field in the indicated direction. Claim 6 A method for manipulating a sound field according to claim 1, wherein the at least one effect parameter value includes an indicated direction, and the step of applying the first effect includes the step of using the at least one effect parameter value to increase the acoustic level of the sound field in the indicated direction compared to the acoustic level of the sound field in other directions. Claim 7 A method for manipulating a sound field according to claim 1, wherein the at least one effect parameter value includes an indicated direction, and the step of applying the first effect includes the step of using the at least one effect parameter value to reduce the acoustic level of the sound field in the indicated direction compared to the acoustic level of the sound field in other directions. Claim 8 A method for manipulating a sound field according to claim 1, wherein the at least one effect parameter value indicates a position within the sound field, and the step of applying the first effect includes the step of using the at least one effect parameter value to convert a sound source to the indicated position. Claim 9 A method for manipulating a sound field according to claim 1, wherein the at least one effect parameter value includes an indicated direction, and the step of applying the first effect includes the step of using the at least one effect parameter value to increase the directivity of at least one of the sound source of the sound field or the region of the sound field compared to another sound source of the sound field or the region of the sound field. Claim 10 A method for manipulating a sound field according to claim 1, wherein the step of applying the first effect includes the step of applying a matrix transformation to the sound field description. Claim 11 A method for manipulating a sound field according to claim 10, wherein the matrix transformation comprises at least one of rotation of the sound field and translation of the sound field. Claim 12 A method for manipulating a sound field, wherein the sound field description comprises a hierarchical set of basis function coefficients in claim 1. Claim 13 In claim 1, the sound field description comprises a plurality of audio objects, a method for manipulating a sound field. Claim 14 delete Claim 15 A device for manipulating a sound field, comprising: receiving a bitstream including metadata and a sound field description, wherein the metadata includes a first effect identifier specifying a type of first effect to be applied to the sound field description and at least one effect parameter value defining how to apply the first effect, and a second effect identifier specifying a type of second effect to be applied to the sound field description; a decoder configured to parse the metadata to obtain the first effect identifier, the second effect identifier, and the at least one effect parameter value, and to determine not to apply the second effect identified by the second effect identifier to the sound field description; and a renderer configured to apply the first effect identified by the first effect identifier to the sound field description. Claim 16 In claim 15, a device for manipulating a sound field further comprising a modem, wherein the modem receives a signal representing the bitstream; and is configured to provide the bitstream to the decoder. Claim 17 A device for manipulating a sound field, comprising: a memory configured to store a bitstream including metadata and a sound field description, wherein the metadata includes a first effect identifier specifying a type of first effect to be applied to the sound field description and at least one effect parameter value defining how to apply the first effect, and a second effect identifier specifying a type of second effect to be applied to the sound field description; and a processor coupled to the memory, wherein the processor parses the metadata to obtain the first effect identifier, the second effect identifier, and the at least one effect parameter value; determines not to apply the second effect identified by the second effect identifier to the sound field description; and is configured to apply the first effect identified by the first effect identifier to the sound field description. Claim 18 A device for manipulating a sound field according to claim 17, wherein the processor is configured to parse the metadata to obtain a timestamp corresponding to the first effect identifier and to apply the first effect by using the at least one effect parameter value to apply the first effect to a portion of the sound field description corresponding to the timestamp. Claim 19 A device for manipulating a sound field according to claim 17, wherein the processor is configured to combine the at least one effect parameter value with a user command to obtain at least one revised parameter. Claim 20 A device for manipulating a sound field according to claim 17, wherein the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the first effect by using the at least one effect parameter value to rotate the sound field in the indicated direction. Claim 21 A device for manipulating a sound field according to claim 17, wherein the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the first effect by using the at least one effect parameter value to increase the acoustic level of the sound field in the indicated direction relative to the acoustic level of the sound field in other directions. Claim 22 A device for manipulating a sound field according to claim 17, wherein the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the first effect by using the at least one effect parameter value to reduce the acoustic level of the sound field in the indicated direction compared to the acoustic level of the sound field in other directions. Claim 23 A device for manipulating a sound field according to claim 17, wherein at least one effect parameter value indicates a location within the sound field, and the processor is configured to apply the first effect by using the at least one effect parameter value to convert a sound source to the indicated location. Claim 24 A device for manipulating a sound field according to claim 17, wherein the at least one effect parameter value includes an indicated direction, and the processor is configured to apply the first effect by using the at least one effect parameter value to increase the directivity of at least one of the sound source of the sound field or the region of the sound field compared to another sound source of the sound field or the region of the sound field. Claim 25 A device for manipulating a sound field according to claim 17, wherein the processor is configured to apply the first effect by using the at least one effect parameter value to apply a matrix transformation to the sound field description. Claim 26 A device for manipulating a sound field according to claim 25, wherein the matrix transformation comprises at least one of rotation of the sound field and translation of the sound field. Claim 27 In claim 17, the above sound field description is a device for manipulating a sound field comprising a hierarchical set of basis function coefficients. Claim 28 In claim 17, the sound field description is a device for manipulating a sound field, comprising a plurality of audio objects. Claim 29 delete Claim 30 In claim 17, the device is a device for manipulating a sound field, comprising a custom integrated circuit including the processor.

Images