Metadata for spatial audio rendering
By using metadata to control spatial audio rendering on the decoding side during playback of the sound program, the problem of insufficient rendering of audio objects orientation and orientation in the prior art is solved, and high spatial resolution and realistic audio rendering effects are achieved.
Patent Information
- Application Number
- CN202410695717.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-15
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to achieve high spatial resolution audio rendering during playback of sound programs, especially in controlling the orientation and orientation of audio objects, and lacks effective methods for applying scene reverb and post-processing reverb.
Control the spatial audio rendering process on the decoding side by using metadata, including decoding audio objects from the bitstream and applying scene reverb, post-process reverb, and radiation patterns based on the metadata to achieve precise position and orientation of the audio objects Render.
High spatial resolution rendering of sound programs is realized, which enhances the listener's perceptual experience, makes the position and direction of audio objects in 3D space more realistic, and improves the rendering accuracy and flexibility of audio objects.
Smart Images

Figure CN119071713A_ABST
Abstract
Description
[0001] Related Applications
[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 505,965, filed on June 2, 2023, which is incorporated herein by reference. Technical Field
[0003] One aspect of the disclosure herein relates to spatial audio rendering of a sound program by a decoder-side process according to metadata associated with the sound program, wherein the metadata is provided by a process performed in a content creation side. Other aspects are also described. Background Art
[0004] The sound program may be produced as a live recording, such as a recording of a concert or sporting event (with or without accompanying video), or it may be pre-recorded or pre-authored, for example, using a software application development platform, such as a soundtrack for a segment of a video game. In all cases, digital signal processing may be used on the content creation side to adjust the sound program according to the taste of the content creator (e.g., someone working as an audio mixer). The adjusted sound program may then be encoded to reduce the bit rate before being delivered to the listener's playback device, for example, over the Internet. At the playback device, or on the decoding side, the sound program is decoded and then rendered into speaker driver signals suitable for the listener's sound subsystem (e.g., headphones, surround sound loudspeaker arrangement).
[0005] The sound program may be digitally processed by a spatial audio renderer so that the resulting speaker driver signals produce a listening experience in which the listener perceives the program closer to how they would hear the scene if it were present in the scene being recorded or synthesized. A spatial audio renderer would enable the listener to, for example, perceive the sound of a bird chirping as coming from a few meters to their right, and the rustling of another animal through leaves on the ground coming from a few meters to their left, or the sound of wind blowing through trees as surrounding them. Summary of the invention
[0006] A content creator may decide to make a sound program so that it carries perceptible sound cues for a particular scene. For example, in one case, the creator may want the listener to perceive a person's voice as if the person was standing at the bottom of a canyon while the listener was at the edge of the canyon. In another case, the person should sound as if they were only a few feet away from the listener and in a small office. These different perceptions can be achieved by digitally processing the speech signals for spatial audio rendering of the sound program. Various aspects of the present disclosure enable the content creation side to control how the discrete audio objects that make up the sound program are rendered by the decoding side to achieve high spatial resolution, while enabling the decoder side to also control the rendering process to treat the position and orientation of the object as a virtual sound source relative to the listener. One aspect relates to controlling the reverberation effect when rendering an audio object. Another aspect relates to applying a directivity function or radiation pattern based on the position and orientation of the listener relative to the object or the position of the viewing direction when rendering an audio object from the listener's viewpoint.
[0007] According to one aspect of the present disclosure, a decoding-side method for spatial audio rendering using metadata includes: decoding several audio objects, several channels or HOA representations of a sound program from a bitstream; and receiving metadata of the sound program, wherein the metadata includes a first message indicating whether the decoding-side processing during playback of the sound program is: i) based on each audio object or based on a group of multiple objects from several audio objects; ii) based on each channel or based on a group of multiple channels from several channels; or iii) based on the HOA representation to apply scene reverberation.
[0008] In one aspect, the metadata further includes an index to an impulse response (IR) selected from a plurality of impulse responses (IRs), the method further including: not applying scene reverb during playback according to the instruction in the first message; or applying scene reverb during playback according to the index specified in the first message. In another aspect, the metadata further includes an index to a set of reverb parameters, the method further including: not applying scene reverb during playback in response to the first message; applying scene reverb during playback according to the index to the set of reverb parameters specified in the first message; or applying scene reverb during playback according to the IR modified by the reverb parameters specified in the first message. In another aspect, the metadata includes a full room geometry or an index to a selected set of room geometry-based reverb parameters, the method further including: not applying scene reverb during playback according to the first message; applying scene reverb during playback according to the full room geometry as indicated by the first message; or applying scene reverb during playback according to the selected set of room geometry-based reverb parameters as indicated by the first message.
[0009] In one aspect, the metadata further includes a second message, the method further including: not applying post-processing reverb after scene reverb during playback according to the second message; or applying post-processing reverb according to a default acoustic environment after scene reverb during playback as indicated by the second message. In another aspect, the method further includes: not applying post-processing reverb as indicated by the second message; applying post-processing reverb according to a default acoustic environment as indicated by the second message; or applying post-processing reverb according to shortened IR or early reflection IR only as indicated by the second message.
[0010] In one aspect, the metadata includes reverberation parameters such as reverberation parameters based on room geometry, the method further comprising: not applying scene reverberation according to the first message; or applying scene reverberation according to the reverberation parameters. In another aspect, the metadata includes a set of filter coefficients, the method further comprising: not applying scene reverberation according to the first message; or applying scene reverberation according to a reverberation digital filter defined by the set of filter coefficients.
[0011] According to another aspect of the present disclosure, a method for spatial audio rendering using metadata performed by a decoding side includes: decoding an audio object of a sound program from a bitstream; and receiving metadata of the sound program, wherein the metadata includes a first message instructing the decoding side as to which radiation pattern from a set of radiation patterns stored on the decoding side to apply to the audio object during spatial audio rendering for playback of the sound program.
[0012] In one aspect, the first message includes a first index indicating the type of the audio object and a second index indicating the viewing direction or orientation of the audio object. In another aspect, the method according to claim 10 further includes identifying a selected one of the radiation patterns based on the first index and the second index; and using the selected one of the radiation patterns to perform spatial audio rendering on the audio object. In some aspects, the method also includes obtaining the position of the listener during playback, wherein the spatial audio rendering is further based on the position of the listener. In another aspect, the method also includes decoding another audio object of the sound program from the bitstream, wherein the metadata also includes instructions to the decoding side to apply one of the following items contained in the metadata to the other audio object during playback of the spatial audio rendering to generate an audio object with a radiation pattern: a set of digital filter coefficients defining an impulse response; a set of HOA coefficients based on each frequency band; or a set of radiation patterns based on each frequency band. In another aspect, each of the radiation patterns includes a shape of omnidirectional, cardioid, hypercardioid, or dipole.
[0013] According to another aspect of the present disclosure, an electronic device includes: at least one processor; and a memory having instructions stored therein, which, when executed by the at least one processor, causes the electronic device to: decode several audio objects, several channels, or HOA representations of a sound program from a bitstream; and receive metadata of the sound program, wherein the metadata includes a first message indicating whether a decoding side processes during playback of the sound program: i) based on each audio object or based on a group of multiple objects from several audio objects; ii) based on each channel or based on a group of multiple channels from several channels; or iii) based on the HOA representation to apply scene reverberation.
[0014] In one aspect, the metadata further includes an index to an impulse response (IR) selected from a plurality of impulse responses (IRs), wherein the memory has further instructions for: not applying scene reverb during playback according to the instructions in the first message; or applying scene reverb during playback according to the index specified in the first message. In another aspect, the metadata further includes an index to a set of reverb parameters, wherein the memory has further instructions for: not applying scene reverb during playback in response to the first message; applying scene reverb during playback according to the index to the set of reverb parameters specified in the first message; or applying scene reverb during playback according to the IR modified by the reverb parameters specified in the first message. In some aspects, the metadata includes the complete room geometry or an index to a selected set of reverberation parameters based on the room geometry, and the memory has further instructions for: not applying scene reverberation during playback in accordance with the first message; applying scene reverberation during playback in accordance with the complete room geometry as indicated by the first message; or applying scene reverberation during playback in accordance with the selected set of reverberation parameters based on the room geometry as indicated by the first message.
[0015] In one aspect, the metadata further includes a second message, the memory having further instructions for: not applying post-processing reverb after scene reverb during playback according to the second message; or applying post-processing reverb according to a default acoustic environment after scene reverb during playback as indicated by the second message. In another aspect, the memory has further instructions for: not applying post-processing reverb as indicated by the second message; applying post-processing reverb according to a default acoustic environment as indicated by the second message; or applying post-processing reverb according to shortened IR or early reflection IR only as indicated by the second message. In another aspect, the metadata includes reverb parameters such as reverb parameters based on room geometry, the memory having further instructions for: not applying scene reverb according to the first message; or applying scene reverb according to the reverb parameters.
[0016] According to one aspect of the present disclosure, a method includes: displaying a user interface (UI) of a three-dimensional (3D) acoustic environment including a sound source; receiving, via an input device, a user selection of a location within the UI toward which the sound source is to be oriented within the 3D acoustic environment; determining a set of parameters defining the orientation of the sound source within the 3D acoustic environment toward the location; determining a movement of the location or the sound source within the 3D acoustic environment; and updating the set of parameters to maintain the orientation of the sound source toward the location without user intervention and based on the movement.
[0017] In one aspect, the user selection is a first user selection, wherein the method further comprises receiving a second user selection of a UI item via an input device, the second user selection of the UI item locking the orientation of the sound source to the position, wherein the set of parameters is determined in response to receiving the second user selection. On the other hand, the sound source is a first sound source, and the position is one of: a static position within a 3D acoustic environment; a second sound source within the 3D acoustic environment; or a point within an area surrounded by a set of sound sources within the 3D acoustic environment. In some aspects, the movement of the position comprises movement of the second sound source from the position to a new position within the 3D acoustic environment, wherein the set of parameters is updated so that the orientation of the first sound source follows the second sound source from the position to the new position.
[0018] In one aspect, the sound source comprises a radiation pattern of sound of the sound source, wherein the orientation is a viewing direction of the radiation pattern from the sound source and toward the location. In another aspect, the method further comprises fixing the orientation of the sound source toward the location in response to receiving a user input, such that the orientation of the sound source remains pointed toward the location regardless of movement of the location or the sound source. In another aspect, the method further comprises: encoding the sound source into a bitstream for transmission to a playback device; and providing metadata comprising the set of parameters, wherein the playback device will be configured to render the sound source according to the set of parameters. In another aspect, the method further comprises providing an updated set of parameters after providing the metadata, such that the playback device can adjust the orientation of the sound source during rendering of the sound source.
[0019] According to another aspect of the present disclosure, an electronic device is included, comprising: at least one processor; and a memory having instructions stored therein, which, when executed by the at least one processor, causes the electronic device to: receive an audio object of a sound program, the audio object comprising a radiation pattern for projecting sound toward a viewing direction of the audio object within a three-dimensional (3D) scene; assign a point within the 3D scene to which the viewing direction of the audio object will be directed; determine one or more orientation parameters for the viewing direction of the audio object; determine whether there is movement of a point or an audio object within the 3D scene; and in response to determining that there is movement of the point or the audio object, update the one or more orientation parameters based on the movement and without user intervention to maintain a viewing direction pointing to the point.
[0020] In one aspect, the electronic device further comprises a display, wherein the memory has additional instructions for a user interface (UI) for displaying a 3D scene on the display, wherein the instructions for assigning points include instructions for receiving a user selection of a point within the UI via an input device. On the other hand, the audio object is a first audio object, wherein the point is one of the following: a static position relative to an origin within the 3D scene; a second audio object within the 3D scene; or a position within an area surrounded by a group of audio objects within the 3D scene. In one aspect, the position is the centroid of the area. On the other hand, the sound program is a live audio broadcast. On the other hand, the sound program is a live audio communication between an electronic device and a playback device.
[0021] According to another aspect of the present disclosure, a method performed by a playback device is included, the method comprising: decoding an audio object of a sound program from a bitstream; and receiving metadata of the sound program, wherein the metadata instructs a decoding side to lock a viewing direction of a radiation pattern of the audio object toward a point within a three-dimensional (3D) scene or an audio object as the point moves during playback of the sound program.
[0022] In one aspect, the point is a location within a 3D scene, and the metadata includes coordinates of the location in a coordinate system relative to an origin of the 3D scene. In another aspect, the audio object is a first audio object, wherein the metadata includes a reference to a second audio object, and the point is a location of the second audio object within the 3D scene. In some aspects, the metadata includes a reference to a set of audio objects within the 3D scene, wherein the method further includes: determining the location of the set of audio objects based on the set of references; determining the location of the point within an area defined by the locations of the set of audio objects; and determining a viewing direction of a radiation pattern of the audio object from the location of the audio object to the location of the point within the 3D scene.
[0023] In one aspect, the point is at a first location within the 3D scene, wherein the method further comprises: receiving additional metadata indicating that the point has moved to a second location within the 3D scene; and moving a viewing direction of the radiation pattern from the first location to the second location within a time period. In another aspect, the method further comprises determining an angle between the first location and the second location relative to a location of the audio object within the 3D scene, wherein moving the viewing direction comprises rotating the viewing direction along the angle around the location of the audio object at a speed based on the time period.
[0024] In one aspect, the metadata also includes one or more angular offsets, wherein the method also includes spatially rendering the audio object so that the viewing direction of the radiation pattern is offset from the point based on the one or more angular offsets. In another aspect, the method also includes: determining the position of the listener within the 3D scene; and generating a set of spatially rendered audio signals by spatially rendering the sound program according to the viewing direction of the radiation pattern of the audio object and the position of the listener; and using the set of spatially rendered audio signals to drive a set of speakers to play back the sound program. In another aspect, the point within the 3D scene is the position of the listener, wherein the method also includes determining one or more orientation parameters for orienting the viewing direction of the radiation pattern toward the position of the listener, wherein the set of spatially rendered audio signals is generated according to the one or more orientation parameters.
[0025] According to one aspect of the present disclosure, an encoding-side method for spatial audio rendering using metadata is included, the method comprising: encoding a set of audio objects, a set of channels, or an HOA representation of a sound program into a bitstream; and providing metadata of the sound program, wherein the metadata includes a first message indicating to a decoding-side processing whether to apply scene reverberation during playback of the sound program: i) based on each audio object or based on a set of multiple objects from the set of audio objects; ii) based on each channel or based on a set of multiple channels from the set of channels; or iii) based on the HOA representation.
[0026] In one aspect, the metadata further includes an index to an impulse response (IR) selected from a set of IRs, and the first message indicates a decoding-side processing: not applying scene reverberation; or applying scene reverberation according to the index. In another aspect, the metadata further includes an index to a set of reverberation parameters, and the first message indicates a decoding-side processing: not applying scene reverberation; applying scene reverberation according to the index to the set of reverberation parameters; or applying scene reverberation according to an IR modified by the reverberation parameters. In another aspect, the metadata includes a complete room geometry or an index to a selected set of room geometry-based reverberation parameters, and the first message indicates a decoding-side processing: not applying scene reverberation; or applying scene reverberation according to the complete room geometry; applying scene reverberation according to a selected set of room geometry-based reverberation parameters.
[0027] In one aspect, the metadata also includes a second message indicating whether the decoding side processing applies post-processing reverberation during playback to: not apply post-processing reverberation; or apply post-processing reverberation according to a default acoustic environment. On the other hand, the second message indicates whether the decoding side processing: does not apply post-processing reverberation; applies post-processing reverberation according to a default acoustic environment; or applies post-processing reverberation according to a shortened IR or early reflection IR only. On the other hand, the metadata includes reverberation parameters such as reverberation parameters based on room geometry, and the first message indicates the decoding side processing: does not apply scene reverberation; or applies scene reverberation according to the reverberation parameters. On the other hand, the metadata includes a set of filter coefficients, and the first message indicates the decoding side processing: does not apply scene reverberation; or applies scene reverberation according to a reverberation digital filter defined by the set of filter coefficients.
[0028] According to another aspect of the present disclosure, an encoding side method for spatial audio rendering using metadata includes: encoding an audio object of a sound program into a bitstream; and providing metadata of the sound program, wherein the metadata includes a first index and a second index to a lookup table of a set of radiation patterns stored in a decoding side, wherein the first index and the second index together point to a selected one of the set of radiation patterns, and the metadata indicates that the decoding side applies the selected one of the set of radiation patterns to the audio object during playback spatial audio rendering of the sound program.
[0029] In one aspect, the first index indicates the type of object, and the second index indicates the direction in which the main axis of the object of that type is located. On the other hand, the second index points to a yaw, pitch, and roll triplet or quaternion. On the other hand, the method also includes: encoding another audio object of the sound program into the bitstream, wherein the type of the other object is not in the lookup table; and providing one of the following items together with instructions in the metadata to the decoding side to apply the one of the following items to the other audio object during playback spatial audio rendering: a set of digital filter coefficients defining an impulse response; a set of HOA coefficients based on each frequency band; or a set of radiation patterns based on each frequency band. On the other hand, each of the radiation patterns includes an omnidirectional, cardioid, hypercardioid, or dipole shape.
[0030] In one aspect, the sound source in the viewing direction may include an avatar of the user within a 3D acoustic environment (e.g., an XR environment), and the orientation being locked may be the forward-facing direction of the avatar. In one aspect, the movement of a point (position) may include movement of a second audio object from one position to a new position within the 3D environment, wherein one or more parameters are updated so that the viewing direction of the first audio object follows the second audio object from the position to the new position. In another aspect, the viewing direction of the first audio object follows the second audio object so that the radiation pattern rotates along at least one axis of the first audio object when the viewing direction follows the second audio object. In some aspects, the sound program and metadata including the updated orientation parameters may be encoded into the bitstream for transmission to one or more playback devices.
[0031] According to one aspect of the present disclosure, a method performed on an encoding side includes: encoding a sound program into a bitstream, the sound program having an audio object within a 3D scene; and encoding metadata of the sound program into the bitstream, the metadata indicating a position within the 3D scene, when the position or the audio object moves within the 3D scene during spatial audio rendering of the sound program, the radiation pattern of the audio object will remain oriented toward the position.
[0032] In one aspect, the metadata includes several coordinates of the position, which are coordinates of a coordinate system about the origin of the 3D scene. On the other hand, the audio object is a first audio object, and the metadata includes a reference to a second audio object at the position within the 3D scene, so that when any audio object moves within the 3D scene during the spatial audio rendering of the sound program, the radiation pattern will remain oriented toward the second audio object. On the other hand, the metadata includes several references to several audio objects at different positions within the 3D scene, and the position is based on the multiple positions. In some aspects, the position is the centroid of the area defined by the position of the audio object within the 3D scene. On the other hand, encoding metadata includes adding audio object metadata, which includes a reference to metadata, which indicates that the decoding side process locks the direction of the radiation pattern to the position during the spatial audio rendering of the sound program. In one aspect, the metadata also includes one or more angle offsets, which keep the viewing direction of the radiation pattern offset from the position at the position or when the audio object moves within the 3D scene during the spatial rendering of the sound program.
[0033] In one aspect, the location based on the number of objects may be a centroid of an area defined by the location of the audio object within the 3D scene. In another aspect, the metadata may be viewing direction metadata, wherein the playback device may receive audio object metadata including a reference to the viewing direction metadata, wherein the playback device may lock the viewing direction of the radiation pattern in response to receiving the audio object metadata.
[0034] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced by all suitable combinations of the various aspects summarized above and disclosed in the detailed description below and specifically pointed out in the claims section. Such combinations may have advantages not specifically set forth in the above summary of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Various aspects of the present disclosure are described by way of example and are not limited to the illustrations of the various drawings, in which similar reference numerals indicate similar elements. It should be noted that reference to "one" or "an" aspect in the present disclosure is not necessarily the same aspect, and it means at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a given drawing may be used to illustrate features of more than one aspect of the present disclosure, and for a given aspect, not all elements in the drawing may be required.
[0036] Figure 1 is a block diagram of encoding-side and decoding-side processing for spatial audio rendering using metadata with indices to room impulse responses and optionally reverberation parameters.
[0037] Figure 2 Depicted is a block diagram of an exemplary decode-side spatial audio rendering process with reverberation.
[0038] Figure 3 is a block diagram of encoding-side and decoding-side processing for spatial audio rendering using metadata with complete room geometry or full impulse responses for reverberation.
[0039] Figure 4 This diagram is a diagram of encoding-side processing and decoding-side processing for spatial audio rendering using metadata that instructs the decoding side to use a selected directivity.
[0040] Figure 5 An example of viewing directions of an audio object according to one aspect is shown.
[0041] Figure 6 An exemplary dictionary or codebook for storing radiation or directivity patterns is depicted.
[0042] Figure 7 Several stages of an example of a three-dimensional (3D) scene of a sound program according to one aspect are shown.
[0043] Figure 8 is a block diagram of an audio system including at least an audio authoring device according to one aspect.
[0044] Fig. 9 is a flow chart of one aspect of a process at a media content creation device for automatically fixing the orientation of an audio object towards a position within a 3D scene.
[0045] Fig.10 Several stages of a user interface are shown, through which a user can select a position within a 3D scene towards which an audio object remains fixed while movement within the 3D scene occurs.
[0046] Fig.11 is a block diagram of an audio system including an encoding side that locks the viewing direction of a radiation pattern to a point within a 3D scene and provides metadata to a decoding side to spatially render a sound program according to one aspect.
[0047] Fig.12 is a flow chart of one aspect of a process at the encoding side for encoding metadata indicating a location within a 3D scene towards which the radiation pattern of an audio object remains oriented during playback of a sound program.
[0048] Figures 13a to 13c Includes metadata that instructs the decoding side how to direct the radiation pattern of the audio object during playback of the sound program.
[0049] Fig.14is a flow chart of one aspect of a process at a decoding side for decoding metadata that instructs the decoding side to direct a radiation pattern of an audio object toward a point within a 3D scene during playback of a sound program.
[0050] Fig.15 An example of system hardware is shown. DETAILED DESCRIPTION
[0051] refer to Figure 1 , which is a block diagram of exemplary digital processing performed on the encoding side and the decoding side for spatial audio rendering using metadata. On the encoding side, the sound program can be obtained in the form of one or more constituent audio stems, or in the form of one or more constituent audio objects including, for example, a group of two or more objects, each of which is an audio signal, such as a pulse code modulated (PCM) audio signal. Alternatively, the sound program can be in the form of channels (e.g., 5.1 or 7.1.4), or in the form of a high-order ambisonics (HOA). The sound program can also be in a mixed scene format, for example, where the sound field includes not only audio objects, but also channels and HOA representations. Although the techniques described below are in the context of objects, when the sound program is a segment having a channel format or contains segments having a channel format, these techniques can also be applied to each channel, and to the segments of the sound program in the HOA representation.
[0052] A sound program may be music such as the sounds of several instruments played by an orchestra, dialogue such as the individual voices of one or more actors in a play or participants of a podcast, a soundtrack for a movie with dialogue, a music and effects compilation, etc. A sound program may be a live recording (recorded in real time) of, for example, a concert, a sporting event, a live news report, etc., a combination of live recorded and synthesized audio signals, or it may be a pre-recorded or pre-composed music or audiovisual work such as a video game or movie, for example, using a software development kit.
[0053] reverberation
[0054] The content creator (e.g., an audio mixer, who may be a person trained in audio mixing) determines the scene reverberation on a per-object, per-channel, or HOA basis. In one case, the content creator does this by selecting an impulse response (IR) from a dictionary or codebook of IRs. The selected IR is applied to the object, channel, or HOA representation by decoding-side processing during playback. For the purpose of reducing the bit rate, the audio objects of the sound program are encoded into the bitstream, while an index or pointer to the selected IR for each object is provided as metadata associated with the sound program. The metadata may also contain other information about each object, such as the orientation of the object relative to the origin (e.g., its distance from the origin, its azimuth, and its elevation) that can represent the listening orientation. The metadata may also be encoded separately for bit rate reduction. The metadata may be provided to the decoding side via a separate communication channel, or it may be incorporated into the bitstream together with the sound program.
[0055] On the decoding side, the bitstream and metadata are provided to or obtained by the playback device, for example, via the Internet. The playback device can be, for example, a digital media player in a console, smart phone, tablet computer, etc. One or more decoding-side processes are performed by a programmed processor in the playback device. These are used to undo the encoding to restore the audio objects, channels, or HOA representations that make up the sound program, and to extract an index to the selected IR from the metadata. The same or another decoding-side process can then use the extracted index to perform a lookup in a dictionary or codebook of the IR to obtain the selected IR. Note here that the IR can be defined by its time domain behavior, its transfer function, or by digital filter coefficients, which in turn define a reverberant digital filter whose impulse response (or equivalently, its transfer function) is the selected IR.
[0056] The decoding side further includes a spatial audio renderer 101 . Figure 2A block diagram depicting an exemplary spatial audio rendering process that produces speaker driver signals 109 for a sound subsystem suitable for a listener depicted by a speaker symbol. The speaker driver signals 109 may be binaural or headphone signals, or they may be speaker driver signals for a particular type of surround sound subsystem. The spatial audio renderer does this by applying its corresponding selected IR to each of the extracted or decoded audio objects 103 (audio signals) of the sound program, and then converting the resulting scene reverberation audio object 105 into the speaker driver signals 109. As an example, the conversion may be performed by first converting the decoded audio objects 103 into a high-order ambisonics HOA format and then converting the HOA format into the speaker driver signals 109. Alternatively, the decoded audio objects 103 may be converted directly into the channel format of the sound subsystem. The speaker driver signals 109 will then drive the sound subsystem that reproduces the sound program.
[0057] The output produced by the sound subsystem enables the listener to experience the sound program desired by the content creator with fine granularity or high spatial resolution, because each audio object, channel or HOA fragment is now rendered in a discrete manner with its corresponding scene reverberation effect. In addition, if the spatial audio renderer 101 is so equipped, each audio object and its corresponding scene reverberation can be rendered spatially based on the position (e.g., distance and direction) of the listener relative to any appropriate reference, which also takes into account the position and orientation (viewing direction) of the virtual sound source that is regarded as the source of the sound defined in the audio object. Finally, in some cases, such as when there are not many audio objects, there can be encoding advantages relative to encoding the sound program as multi-channel. These advantages can offset the increased complexity of decoding-side processing in the playback device.
[0058] Figure 2 Yet another aspect is shown in which the encoding side processing provides an index to one or more parameters specified by the content creator for modifying the selected scene reverberation IR as part of the metadata. The modification may be performed by the spatial audio renderer 101 on the decoding side. Parameters that may be stored in a codebook accessible to both the encoding side and the decoding side may include pre-delay, RT60, wet-dry mix, reverberation width, high frequency damping cutoff, a flag for specifying only early reflections, early reflection levels and times, and late reverberation levels. This feature enables an IR dictionary or codebook to have a smaller number of predetermined IRs, while enabling greater flexibility in determining the desired scene reverberation, as content creators can now modify the IRs of their choice in several ways.
[0059] Figure 2Another aspect is shown, wherein the spatial audio renderer 101, if so equipped, can apply post-processing reverb to identified audio scene components (e.g., audio objects, channels, or HOA representations) of a sound program during playback, thereby producing a post-processing reverberated audio object 107. This would be in addition to applying the scene reverb discussed above (which produces the scene reverberation audio object 105). The post-processing reverb can also be in the form of an IR convolved with the respective scene reverberation audio object. The post-processing IR incorporates the acoustic properties of the listener's actual room into the sound program, and therefore it can enhance the listener's sense of externalization of the audio object as a virtual sound source. In other words, the sound of the audio object is more likely to be perceived as coming from outside the listener's head.
[0060] In one aspect, the post-processing IR is controlled by metadata, allowing the content creator to choose between a previously determined IR for a default acoustic environment (e.g., as a function of, for example, the type of playback device), or an IR generated based on sensing the environment or room that the listener is experiencing the playback in. The latter IR may be generated based on measuring the dimensions of a real room (e.g., the room surrounding the listener at playback time), sensing data of certain objects in the real room such as furniture and windows, or acoustic parameters such as reverberation time RT60 and direct to reverberant ratio.
[0061] Now go to Figure 3 , which is a block diagram of another arrangement of processing for spatial audio rendering using metadata performed on the encoding side and the decoding side. Figure 3 In the sound program (similar to Figure 1 ) is obtained in the form of one or more audio groups, or one or more audio objects that may include a group of two or more objects, where each object or group, or grouped objects or groups, is an audio signal, such as a pulse code modulated (PCM) audio signal. The content creator in this case selects or specifies the complete room geometry or full IR for his desired scene reverberation (to be applied by the decoding side), rather than selecting from a dictionary or codebook of predetermined IRs (see also Figure 2). The complete room geometry or full IR is provided in the metadata, and the metadata is then provided to the decoding side either separately from the sound program, or incorporated into the bitstream containing the sound program. The complete room geometry is extracted by the decoding side process and then used by the spatial audio renderer 101 to calculate (simulate or synthesize) the IR based on the complete room geometry, and then apply the synthesized IR to the extracted audio object (or group of audio objects) or grouping, as indicated in the metadata. The complete room geometry may include the dimensions of the room, the materials of the room's walls, floor, and ceiling, one or more scattering objects such as furniture, and the materials of the scattering objects. In the event that the metadata instructs the decoding side to use the complete IR in the metadata, the decoding side will extract the complete IR and apply it directly to the extracted audio object or grouping as specified in the metadata.
[0062]
[0063] Table 1
[0064] Table 1 shows an exemplary set of reverberation-related fields in metadata.
[0065] Table 1 shows exemplary messages that may be inserted into the metadata by the encoding side process. In particular, the table shows an mSceneReverbIndex message for indicating whether (and how) the decoding side applies scene reverberation, and shows an mPostProcReverbIndex message for indicating whether (and how) the decoding side applies post-processing reverberation. Turning to mSceneReverbIndex, the message may include one or more values for indicating how the decoder handles reverberation for: 1) the global audio scene; 2) individual scenes, such as on a per-audio object basis or on a group of audio objects; 3) on a per-channel basis or on a group of multiple channels; and / or 4) on a HOA representation basis. In one aspect, the index may be static or may be dynamic, such that the parameter (scene reverberation) may change during playback.
[0066] Based on the index, the message within the metadata may indicate how the decoder side applies scene reverb. As shown, when the index is "0", the message indicates that the decoder side does not apply scene reverb, so that the audio rendered by the decoding side has no reverb added. In this case, the output sound of the decoding side may be "completely dry" because the sound may not include any added reverb. When the index is "1", the message indicates that the decoding side applies scene reverb according to one or more preset codebooks with one or more reverb parameters. In particular, the message may indicate that the decoding side will apply reverb according to one or more preset codebooks of IR and / or reverb parameters. In one aspect, the metadata may include an index of an IR from several IRs (e.g., in a codebook), wherein the metadata (or message) may indicate that the scene reverb will be applied according to the index. On the other hand, the metadata may include an index to a set of reverb parameters, which may include one or more of pre-delay, RT60, wet-dry mix, reverb width, high frequency damping cutoff, a flag for specifying only early reflections, early reflection levels and times, and late reverberation levels.
[0067] When the index is "2", the message may indicate to the decoding side that the encoding side will directly transmit one or more reverberation parameters. When the index is "3", the message may indicate to the decoding side that the scene reverberation is applied according to one or more preset codebooks having one or more reverberation filter coefficients. When the index is "4", the message may indicate to the decoding side that the encoding side will directly transmit the reverberation filter coefficients, which may be used by the decoding side to define the reverberation digital filter. In one aspect, the metadata may include a set of filter coefficients from which the decoding side may select and use. In one aspect, the encoding side may transmit lightweight parameters and / or filter coefficients to be used before receiving the reverberation parameters and / or filter coefficients, respectively.
[0068] When the index is "5", the message may instruct the decoding side to apply scene reverberation according to a preset codebook with reverberation parameters based on room geometry. For example, the message may instruct the decoding side to apply scene reverberation according to the complete room geometry or to apply scene reverberation according to a selected set of reverberation parameters based on room geometry. In one aspect, the metadata includes a field for the content creation side to specify the complete room geometry, and another field to specify an index to a selected set of reverberation parameters based on room geometry. In one aspect, the metadata may include geometry-based reverberation parameters, which may include at least one of the following: the number of subbands, RT60 per subband, early room energy per subband, late room energy per subband, room volume, and / or room surface. When the index is "6", the message may instruct the decoding side that the encoding side will guide the transmission of reverberation parameters based on room geometry. In one aspect, the encoding side may also transmit lightweight parameters, which may also be transmitted for use before the reverberation parameters based on room geometry are received by the decoding side.
[0069] Turning to mPostProcReverbIndex, this message (which may be a separate message from the mSceneReverbIndex message) may include one or more values indicating how the decoding side applies post-processing reverb, which may be after scene reverb. In another aspect, post-processing reverb may be applied, while the decoding side may not apply scene reverb. In one aspect, this message may indicate global parameters describing the acoustic environment. In some aspects, the values indicated by the message may include one or more static parameters.
[0070] When the value is "0", the message indicates that the decoding side does not apply post-processing reverb (e.g., completely dry). When the value is "1", the message indicates that the decoding side applies a default post-processing reverb. In particular, the post-processing reverb is applied during playback of the audio content according to the default acoustic environment. When the value is "2", the message indicates that the decoding side applies only early reflections. In one aspect, the message may indicate that the decoding side applies post-processing reverb according to a shortened IR or only early reflection IR. In this case, the metadata may include the length of the early reflections.
[0071] Radiation pattern / directivity and viewing direction
[0072] Now go to Figure 4, which depicts encoding-side and decoding-side processing for spatial audio rendering using metadata, where the metadata instructs the decoding side how to render a specific audio object (as a virtual sound source) as a virtual sound source with a selected directivity during playback of a sound program. Discrete audio objects can be human voices or sounds produced by musical instruments. In one case, a content creator selects a first index into a dictionary or codebook of radiation patterns or directivities stored in the decoding side. The directivity dictionary contains directivities for different types of sound producing objects. Content creators do not need to know the actual radiation pattern they are selecting, but can simply select the first index as the type of their discrete object based on their knowledge of the type of discrete object. Each object type has a corresponding radiation pattern, such as human voice, guitar, grand piano, etc.
[0073] The metadata will also contain the selected viewing direction or orientation of the virtual sound source associated with a given audio object. Figure 5 An example is shown in , which shows an audio scene 500 including a listener (or listener position) 501 and several audio objects 502a and 502b, each with a corresponding viewing direction 503a and 503b. In particular, the viewing direction 503a points to the left of the listener 501, while the viewing direction 503b points away from the listener 501 (and in front of it). For example, the audio objects can be human voices or sounds produced by a specific machine (for example, in a movie or video game). The encoding side processing can determine the viewing direction of the character representing the person or the machine. It then inserts a pointer to the viewing direction as a second index into the metadata. The second index points to a previously defined viewing direction, which can be, for example, in the form of an azimuth-elevation pair, which represents the direction in which the main axis is located (as a yaw-pitch-roll triplet or as a quaternion). Yaw-pitch-roll triplets and quaternions may be used when the orientation of the virtual sound source about the look direction axis is important or significant, such as when the radiation pattern does not have axisymmetry about its look direction.
[0074] Similar to the reverberation aspect described above, the encoding side process repeats to determine the selected directivity and viewing direction for any other desired objects in the sound program. Indexes to the selected directivities (associated with their corresponding audio objects) are provided as metadata associated with the sound program along with the viewing direction. Such metadata enables the decoding side to render the audio objects so that the listener can hear the virtual sound sources (associated with the objects) differently at each of the multiple listening positions in 3D space, because at each listening position, the source directivity filtering depends on the position of the listener.
[0075] On the decoding side, the bitstream and metadata are provided to or obtained by a playback device, for example, via the Internet. The playback device may be, for example, a digital media player in a console, a smart phone, a tablet computer, etc. One or more decoding-side processes are performed by a programmed processor in the playback device. These are used to undo the encoding to restore the audio objects that make up the sound program, and to extract from the metadata an index of the selected directivity of each of the decoded audio objects.
[0076] The decoding side processing may then use the extracted first index (e.g., which may represent the type of object) and the second index (e.g., representing the direction in which the main axis of an object of that type is located) to perform a lookup into a dictionary or codebook of stored directivities to obtain the selected directivity or radiation pattern. As described above, the spatial audio renderer 101 then applies its corresponding selected directivity to each of the extracted or decoded audio objects 103 (audio signals) of the sound program, and then converts the resulting directivity-adjusted audio objects into speaker driver signals. Note here that while all of these aspects may not be mentioned here, the above description may be combined with other aspects. Figures 1 to 3 Several variations of the reverberation described in Figure 4 The directivity aspect of the audio object 103 (e.g., converting the decoded audio object 103 into a higher-order ambisonics HOA format, and then converting the HOA format into a speaker driver signal 109, or directly into a channel format of a sound subsystem).
[0077] Figure 6 It shows how the directivity (radiation pattern) can be stored in a codebook. Each row of the codebook refers to a different object type, and each column refers to a different viewing direction of the object (e.g., azimuth and elevation and / or yaw, pitch and roll). In one aspect, the column can refer to the emission direction of the sound radiation. This is of course just an example, because the two row axes and column axes can be defined differently. Each entry of the codebook can be a set of HOA coefficients for each frequency band of the sound field, and when the object of this type is oriented in the viewing direction given by the column, the sound field will be generated by the object of this type assigned to the row. On the other hand, each entry of the codebook can be a coefficient that will be generated when the listening position is in the position given by the coordinates given by the column in the object reference. Therefore, the decoding side process extracts the object type and determines the coordinates of the listening position in the object reference so as to use this data as an index into the lookup table to obtain the radiation pattern, which is then used by the spatial audio renderer 101 to render the decoded audio object. The output radiation pattern may be applied directly to the audio object, or an interpolation may be performed, for example in a look-up table, between two radiation patterns whose coordinates are around the desired direction.
[0078] refer to Figure 4For greater realism, when the spatial audio renderer 101 renders an object using a radiation pattern obtained from a table lookup, it may also take into account the orientation of the audio object (virtual sound source) relative to the listener's orientation. On the other hand, the audio renderer may also take into account the listener's viewing direction. In one aspect, during the rendering of the audio content, the listener's orientation or the listener's viewing direction may change as a function of time. In this case, if the listener's viewing direction is known (e.g., measured based on head tracking indicated by the listener's orientation) and the object's viewing direction is received by a bitstream (e.g., object metadata), an appropriate radiation pattern may be selected. On the other hand, the selected radiation pattern index (or multiple indices, in the case of interpolation between two responses) may depend solely on the listener's orientation, or more specifically on the orientation of their ears in the object's directional reference.
[0079] In the case where the type of the audio object does not match any of the types of audio objects in the available or stored codebook (or the type of the other object is not in the codebook, lookup table, or dictionary), the content creator may choose to insert a direct definition of the directivity of the non-matching object into the metadata. For example, the definition may include the selected radiation pattern when the non-matching object is oriented at the selected azimuth and elevation angles. Alternatively, the definition may be the HOA coefficients (per band) of the sound field generated by the non-matching object oriented in the selected viewing direction. Thus, during playback spatial audio rendering, metadata may be provided together with one of the following items and instructions to the decoding side for applying the item to another audio object: a set of digital filter coefficients defining an impulse response; a set of HOA coefficients based on each frequency band; or a set of radiation patterns based on each frequency band. Each radiation pattern in the radiation pattern may be given in shapes such as omnidirectional, cardioid, hypercardioid, dipole, etc. In addition, the metadata may indicate the sharpness of the shape. Alternatively, the shape may be a cone defined by a specific inner angle, outer angle, and outer gain. In yet another alternative, the shape may be a physics-based spherical cap given by the sphere radius and the cap opening angle.
[0080]
[0081] Table 2
[0082] Table 2 shows an exemplary set of fields in metadata for controlling the application of radiation patterns.
[0083] Table 2 shows an exemplary set of fields in metadata for controlling the application of radiation patterns. In particular, the table shows a radiation pattern description (RPD) message for instructing a decoding device to apply a reverberation pattern during playback of audio content. The message may include one or more values (indexed), based on which the decoding side may apply the selected reverberation. When the index is "0", the encoding side indicates that the filter coefficients for M viewing directions of the sound object are stored in the decoding side (e.g., a renderer on the decoding side). In this case, based on the selected viewing direction, the renderer may select an appropriate filter coefficient (e.g., perform a table lookup). When the index is "1", the encoding side indicates that the filter coefficients for M viewing directions of the decoding side for O sound source types are stored in the renderer. Therefore, when the type of the object is known, the decoding side can retrieve the appropriate filter coefficients for the appropriate type of object.
[0084] When the index is "2", the message indicates that the decoder side transmits the radiation pattern parameters for M viewing directions of the sound object to the renderer. In one aspect, the radiation pattern parameters may include at least one of the following: a set of cardioids for each subband (e.g., [frequency, mode (0=omnidirectional, 0.5=cardioid, 1=dipole), sharpness (e.g., gain=gain^sharpness)], a set of cones for each subband [frequency, inner angle, outer angle, outer gain], a physically based spherical cap [sphere radius, cap opening angle], a spherical harmonic description of each subband. When the index is "3", the radiation pattern parameters for M viewing directions of O sound source types are transmitted to the decoding side. In this case, the metadata received by the decoding side may include at least one radiation pattern parameter described herein. When the index is "4", the message indicates that the encoding side is to transmit the radiation pattern parameters for M viewing directions of each audio object to the decoding side.
[0085] Thus, the viewing direction of an object (and its index) may be transmitted via the bitstream (e.g., via object metadata). The viewing direction may be determined on the decoding (playback) side, and the viewing direction of the object received via the bitstream metadata may select appropriate filter coefficients / parameters. In one aspect, indices 1 and 4 of Table 2 (the source type index for each object) may be transmitted from the encoding side to the decoding side.
[0086] In one aspect, audio objects can be used in audiovisual media to simulate acoustic characteristics of audio signals in three-dimensional (3D) space, either alone or in combination with any number of object-based, channel-based, and / or ambisonic audio signals. Audio objects in a 3D audio scene can be rendered based on a scene graph (or scene tree structure) that describes their location relative to a listener's stationary or moving position. More about scene graphs is described herein.
[0087] The ability to apply physically modeled or synthetically generated radiation patterns when spatially rendering one or more audio signals simulates the directivity and spectral characteristics of acoustic behavior measured by the radiation pattern. Radiation patterns can be derived from physical models of sound sources (such as the human vocal tract, acoustic musical instruments, speaker box enclosures) and microphone polar patterns that can describe the signal area and spectral sensitivity of different microphone enclosures. When such radiation patterns are applied to spatial rendering of audio signals (e.g., audio object signals), the orientation of the radiation pattern relative to the position of the listener can inform the perceived directionality of sounds emanating from the audio object (sound source) position; for example, human speech directed toward or away from the listener from the same fixed relative position.
[0088] During the content creation of a sound program, the creator may wish to make the orientation of the radiation pattern of an audio object move relative to the listener in the sound space. This may be the case when the orientation (e.g., orientation and / or position in the sound space) of the audio object changes over time. For example, the creation of the sound of a person talking to another person passing by may require that the orientation of the sound source move when the other person passes by. In order to produce this effect, the creator may need to manually adjust the orientation of the radiation pattern of the audio object, such as through a software program (e.g., the user interface through which the creator creates the sound program). This would mean manually adjusting the orientation parameters of the sound source at various times in the sound program, which may be a laborious and inefficient task, especially when there are several audio objects in the sound space that will need to be manually updated by the creator. Therefore, a method and system are needed for fixing (or locking) the viewing direction (or orientation) of an audio object (e.g., its radiation pattern) to a point (or position) in the sound space so as to automatically keep the viewing direction of the radiation pattern at a point in the sound space, regardless of whether the point or the audio object moves in the sound space.
[0089] To solve this problem, the present disclosure provides an audio creation system that effectively and efficiently fixes the orientation of the radiation pattern of the audio object to a point (e.g., a static position, another audio object, etc.) within a 3D acoustic scene or when the audio object moves within the scene. Specifically, such an audio creation system can automatically adjust the viewing direction of the radiation pattern relative to a coordinate system so as to keep the radiation pattern pointing to a point or position in space, regardless of whether the point or object moves in space. An audio software application (such as an audio content creation application executable by an electronic device) can perform operations to fix the orientation of a sound source within a 3D scene of a sound program. For example, a position within a 3D acoustic environment (or acoustic scene) toward which a sound source associated with an audio object will be oriented can be determined within the environment. Specifically, the orientation can be relative to the radiation pattern of the sound source, wherein the radiation pattern can be used to define the acoustic effect of the diffusion of the sound source within the 3D scene. For example, such a position can be user-defined within a user interface (UI) of a software application. A set of parameters that can define the orientation of the sound source toward the position can be determined. Such parameters may include a quaternion, which may be four normalized parameters that define the orientation of the sound source (e.g., relative to the axes of a coordinate system). The system may determine that there is a position or motion of the sound source. The authoring system may update the set of parameters to maintain the orientation of the sound source toward the position without user intervention and based on the motion. Thus, as the orientation of an audio object (or the position to which the object points) changes over time within a sound program, the system may continuously update the parameters that define the orientation vector of the radiation pattern. Thus, the present disclosure provides a method and system by which the parameters that define the orientation vector of a radiation pattern may be continuously and automatically updated within an audio content authoring application to ensure that the orientation of the audio object radiation pattern always points to a position within a 3D scene, which may be, for example, a fixed (static) position, another (moving) audio object in the 3D scene, or a position within an area surrounded by a set of moving audio objects.
[0090] Figure 7Several stages 140 to 142 of an example of a 3D scene 152 of a sound program according to one aspect are shown, wherein the orientation (or viewing direction) of the radiation pattern of audio objects can be automatically updated based on movement within the 3D scene. The first stage 140 shows a 3D scene 152 including a tour bus 143 and pedestrians 147. In one aspect, the 3D scene can be part of a media (sound) program. For example, the scene can be part of an audio / video (A / V) program such as an extended reality (XR) environment, a video game, a movie, a live broadcast, a live communication (e.g., a video conference call), etc., wherein the scene can include a visual (e.g., virtual) environment having the listed elements, and audio objects (audio components) including the environment, wherein a listener at the listener's position within the 3D scene can perceive the sounds of the 3D scene. The 3D scene may include one or more audio objects (sound sources), such as a tour guide 145 who may be speaking to a passenger 146 (both on the tour bus 143), an exhaust vent 144 of the tour bus 143 (whose sound may be heard by a pedestrian 147), and a pedestrian 147 who may be speaking to the passenger 146. In this case, when the 3D scene is rendered spatially (e.g., by a decoding side), the locations of the sound sources may be perceived by a listener based on the listener's location and / or orientation within the 3D scene.
[0091] The second stage 141 shows the radiation pattern of the audio object within the 3D scene 152. Specifically, this stage shows the position of the audio object within the 3D scene, which can include the position and / or orientation of the object within the 3D scene. For example, this stage shows the guide position 150a of the talking guide 145 with the radiation pattern 151a pointing to the passenger position 150d, the exhaust position 150c of the exhaust port 144 of the noise emission with the radiation pattern 151c pointing to the pedestrian position 150b of the pedestrian 147, and the radiation pattern 151b of the talking pedestrian 147 pointing to the passenger position 150d. In this case, each of the dotted arrows can represent the viewing direction of the corresponding radiation pattern that can be directed away from its associated position and toward another position within the 3D scene. In one aspect, the position of the audio object can be relative to one or more reference points such as the global scene origin 148. On the other hand, the viewing direction can indicate the orientation of the corresponding sound source. The orientation can be relative to one or more coordinate system axes within the coordinate system of the 3D scene. For example, the viewing directions include: ψ as the orientation of radiation pattern 151a, λ as the orientation of radiation pattern 151c, and θ as the orientation of radiation pattern 151b. Each of the orientations may be relative to a coordinate axis of a global coordinate system, where global scene origin 148 may be the origin of the system.
[0092] In one aspect, the audio system can determine the viewing direction of the radiation pattern. For example, the creator of the audio content can define a location toward which the viewing direction of the audio object is fixed during playback of the audio content. For example, the creator can define the location through a user interface (UI) of an audio software application, such as Fig.10 As shown. In another aspect, the audio system can automatically define the location (e.g., without user intervention). Once the radiation pattern is pointed at the location, the audio system can determine the orientation (e.g., as an orientation vector) between the sound source and the location within the 3D scene, such as θ for radiation pattern 151b. More information about determining the viewing direction is described herein.
[0093] The third stage 142 shows that the tour bus 143 is driving away from the pedestrian 147. In this case, the audio system can determine whether any viewing direction will be updated based on the movement within the 3D scene to ensure that the viewing direction remains fixed at its corresponding point. In this case, the audio system has determined that the viewing directions of the radiation patterns 151b and 151c will be updated, and then generates new orientations (e.g., orientation parameters) as θ' and λ', respectively, which ensure that the radiation patterns 151b and 151c remain fixed at the orientations 150d and 150b, respectively. As shown, the viewing direction ψ of the radiation pattern 151a has remained the same because the orientation of the tour guide 145 pointing to the passenger 146 has not changed relative to the global reference coordinate axis. Therefore, the audio system may be able to update the viewing direction without requiring the creator to manually perform the update.
[0094] Figure 8 1 is a block diagram of an audio system (hereinafter referred to as "system") 100 that includes at least a media content creation device 600 that locks (or fixes) a viewing direction of a radiation pattern to a point within a 3D scene according to one aspect. The media content creation device (hereinafter referred to as "creation device") 600 can be any type of electronic device that can be capable of creating a media (e.g., video and / or audio) program, such as a desktop computer, a mobile phone, a laptop computer, etc. In one aspect, the creation device can be (or include) an encoding side device capable of encoding a bitstream with media content and / or metadata and transmitting it to a playback (decoding side) device, as described herein.
[0095] The audio creation device may include a network interface 601, a controller 602, a non-transitory machine-readable storage medium (or memory) 603, an input device 604, and a display 605. In one aspect, the device may include fewer or more elements, such as having no input device or including multiple displays. In one aspect, the input device may be separate from the creation device and may be communicatively coupled (e.g., wired or wirelessly connected). The controller 602 may be (or include) a dedicated processor (e.g., one or more processors) such as an application specific integrated circuit (ASIC), a general-purpose microprocessor, a field programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller may be configured to perform the digital signal processing operations and / or networking operations described herein.
[0096] The memory 603 includes an authoring software program 606 and one or more media programs 608, which may include media content 609 and / or metadata 610. In one aspect, a media program may include any type of program such as a movie, a musical composition, an electronic game, etc. A media program may include video content and / or audio content of a piece of media content. In one aspect, the audio content may include one or more acoustic 3D scenes of the media program 608, wherein each scene may include one or more audio objects and / or video content that may correspond to visual elements of the acoustic scene. For example, referring to Figure 7 , video content may include visual representations of an environment such as a tour bus 143 and pedestrians 147 in a bus stop, while audio content may include audio objects associated with sound sources within the environment. Metadata 610 may include information related to the media content. For example, when the media content includes one or more audio objects of a sound program, the metadata may indicate the location and orientation of the object (e.g., orientation parameters such as yaw, pitch, and roll). As shown, the media content and metadata may be stored locally in memory 603. On the other hand, at least a portion of the media program may be stored on an external source such as a remote server.
[0097] Authoring software program 606 may be configured to allow a content creator to author (modify) media programs 608. In this case, the software program may provide a user interface (UI) that may be displayed on display 605 to allow a user to create content by providing user input via input device 604 (e.g., mouse, keyboard, touch-sensitive display). For example, referring to Figure 7, the content creator may cause the tour bus 143 to move from the second stage 141 to the third stage 142 via user input. In one aspect, the authoring software program may allow the user to select one or more parameters (such as the position within the acoustic environment) and the radiation pattern for one or more audio objects within the acoustic environment. The authoring program 606 may also allow the content creator to define the viewing direction of the audio objects within the 3D scene. The authoring program may allow the content creator to create multiple scenes within the media program, each of which may include one or more audio objects. Once authored, the software program may store the media program in a memory, store it at an external source, and / or stream it to one or more other devices for playback.
[0098] In the case of authoring a media program, the authoring software program 606 may set (or define) metadata associated with the media content. For example, the authoring software program may be configured to identify (generate) a set of one or more orientation parameters for an audio object to direct the associated radiation pattern to a point within the 3D scene. In one aspect, the orientation parameters may include any data that may indicate the orientation (viewing direction) of the radiation pattern of the audio object relative to one or more (reference) coordinate axes. For example, Figure 7 In the embodiment of the present invention, the orientation parameters may include an orientation vector θ relative to one or more global coordinate axes of a coordinate system of the 3D scene having a global scene origin 148 at its origin. For example, the orientation parameters may include spherical coordinates within the spherical coordinate system, such as azimuth and / or elevation. In another aspect, the parameters may include one or more quaternions, wherein each quaternion may include four normalized parameters. In another aspect, the orientation parameters may include parameters such as yaw, pitch, and roll, wherein each parameter is associated with a specific axis of the coordinate system of the 3D scene.
[0099] The viewing direction lock 607 of the authoring software program may be configured to determine the viewing direction of one or more audio objects and lock the viewing direction toward a point within the 3D scene to maintain the orientation of the audio object, regardless of the movement of the point or the audio object within the 3D scene. As described herein, the authoring software program may allow a user to select a viewing direction for an audio object. For example, a user may select a point (or object) within the UI of the 3D scene to which the user wishes to lock the viewing direction of the audio object, regardless of the movement of the point or the audio object, via the input device 604. More about selecting a viewing direction is described herein. The viewing direction lock 607 may be configured to determine a point (or position) within the 3D scene to which a sound source associated with an audio object will be directed within the 3D scene, and one or more UI items may be provided that, when selected by the user via the input device 604, may lock the viewing direction toward the point. More about the UI of the authoring software program is described herein.
[0100] The authoring program may be configured to determine a point (or position) within the 3D scene toward which a sound source associated with an audio object will be oriented within the 3D scene. As described herein, this determination may be based on user input through input device 604. On the other hand, user input may be received by other methods, such as through voice commands captured by a microphone of authoring device 600. In either case, the author may define the point at the beginning of the 3D scene of the sound program so that during playback of the 3D scene, the radiation pattern of the sound source remains focused on the point. More content about the author defining the point is described herein. On the other hand, the point may be defined based on media metadata 610 of media program 608. In this case, metadata 610 may specify a position and / or direction within the 3D scene toward which the sound source will remain fixed.
[0101] In another aspect, the viewing direction lock 607 may determine a point toward which a sound source will remain fixed based on a media program 608 currently being created or to be created. In particular, the lock 607 may perform an object recognition algorithm on the media content 609 to determine the viewing direction of the sound source. Figure 7 , the viewing direction lock may determine, based on the angle of the pedestrian's head and / or the pedestrian's gaze, that the pedestrian 147 is looking upward and toward the passenger 146. Thus, the lock may determine that the pedestrian's radiation pattern 151b will be fixed toward the passenger orientation 150d.
[0102] In one aspect, the point towards which the viewing direction of the radiation pattern will remain fixed can be a static location within the 3D scene. Figure 7 , the point may be a location on the ground or in front of the tour bus 143. In another aspect, the point may be another audio object within the 3D scene. This is the case with radiation pattern 151c, which has a viewing direction pointing toward pedestrian position 150b. As another example, the point may be a moving object within the 3D scene, such as the tour bus 143. In another aspect, the audio object associated with the viewing direction may be static or may be moving. For example, referring to Figure 7 , pedestrian 147 may remain stationary within the 3D scene, or pedestrian 147 may move around the environment. In either case, the authoring program may maintain the radiation pattern 151b toward the viewing direction of the passenger position 150d.
[0103] On the other hand, the point may be based on several (static and / or moving) objects within the 3D scene, and may or may not point to any of the specific objects. For example, when there is an audio object, which may be the case when two people are having a one-to-one conversation, the viewing direction of one person may be fixed on the other person (e.g., based on user input). However, in the case where a person is talking to a group of (e.g., speaking) people, the viewing direction of the person who is speaking may point to one of the people in the crowd or based on the position of the crowd. For example, the point may be within an area (or range) surrounded by (audio) objects within the 3D scene. The viewing direction lock 607 may be configured to generate an area based on the position of the audio object within the 3D scene. For example, the lock 607 may generate the area as a polygon, in which each audio object in the audio object is a vertex. In this case, the point within the area may be at the centroid of the area. In one aspect, the area may be generated by an identifier based on a group of objects that may be selected by the user. On the other hand, the area may be user-specified. For example, a user may draw an area within the 3D scene within the UI. On the other hand, the point within the area may be user-specified (e.g., different from the centroid of the area).
[0104] The viewing direction lock 607 can automatically (e.g., without user intervention) update the viewing direction based on (in response to) movement of a point and / or audio object associated with the viewing direction within the 3D scene over a period of time so as to lock the viewing direction of the radial pattern toward the point. In particular, the lock 607 can update the viewing direction during at least a portion of the playback time of the media program 608. In one aspect, the portion of the playback may correspond to movement within the 3D scene. In this case, when the scene changes within the media program (e.g., from a bus stop to a building), the viewing direction can be removed or can remain fixed based on whether the audio object and / or point is within the next scene.
[0105] refer to Figure 7, lock 607 can determine that the tour bus 143 has started to drive away at the third stage 142, and the third stage can be a certain time period after the second stage 141. This determination can be based on an analysis of the media program. For example, the viewing direction lock 607 can determine that the tour bus has moved from its original position based on a visual analysis of the video content. As another example, the lock 607 can determine that the tour bus has moved based on an acoustic analysis of one or more audio objects, where, for example, the movement can be identified by determining that the audio object associated with the speaking tour guide has moved away from the global scene origin 148. On the other hand, the lock 607 can determine the point towards which the viewing direction is moving based on the media metadata 610. For example, when the point is another audio object, the media metadata can indicate that the audio object will move within the 3D scene. When the movement is identified, the viewing direction lock 607 can update the orientation parameters of the viewing direction of the audio object based on the movement so that the viewing direction remains focused on the point. The parameters may be updated by determining a difference between its position relative to the position of the audio object and its new position relative to the original position of the audio object or its original position relative to the new position of the audio object.
[0106] As described so far, the point toward which the viewing direction is locked can be associated with a visual representation of an object within the 3D scene, such as a person or another sound source. On the other hand, the point can be associated with one or more characteristics of the object (or sound source). For example, the orientation of the radiation pattern can follow the direction of the trajectory of an object within the 3D scene. In this case, the viewing direction lock 607 can be configured to determine the trajectory of the object (e.g., based on the metadata of the object and / or based on object recognition), and can be configured to update the viewing direction based on the trajectory (e.g., the direction of the trajectory). On the other hand, the viewing direction lock can determine an orientation parameter relative to the direction of the trajectory of the object. In this way, for example, a radiation pattern associated with the engine of a car can follow the trajectory of the car, or its exhaust pipe can be oriented toward the rear along the trajectory.
[0107] As described thus far, the viewing direction lock 607 can lock the viewing direction toward a point, which can be a static location, a moving object, or a static (or moving) sound source. In another aspect, the viewing direction lock 607 can adjust the point at which the viewing direction is focused on multiple objects. Returning to the previous example of a group of people speaking, some people may move over time while others may remain stationary. In this case, the lock 607 can be configured to generate a new region based on the movement of the moving object, and the point can be adjusted as the centroid of the region changes.
[0108] In one aspect, when the orientation parameter is updated, the viewing direction lock 607 may store the parameter in the metadata 610 so that when the authored media program is rendered spatially, the metadata may be used to define the viewing direction of the radial pattern during playback.
[0109] Fig. 9 160 is a flowchart of one aspect of a process for automatically fixing the orientation of an audio object toward a position within a 3D scene at a media content creation device (e.g., encoding side). In particular, the operations of the process can be performed by an authoring software program 606 (e.g., with its viewing direction locked) being executed by one or more processors of the device 600. The authoring program can receive an audio object of a media program, the audio object having a radiation pattern for projecting sound toward the viewing direction of the audio object within a 3D scene. For example, a content creator can define an audio object within a 3D scene, or the audio object can be an existing sound source within the scene. The process 160 begins by determining a position within a 3D acoustic environment toward which a sound source associated with the audio object will be oriented within the 3D acoustic environment (at block 161). In particular, the authoring software program 606 can be configured to determine a position within the 3D environment at which the viewing direction of the audio object will remain fixed (locked) regardless of movement within the 3D environment. The position can be a point within the 3D environment that can be assigned by the authoring program. In one aspect, the determination (assignment) can be based on a received user input indicating a position. In particular, the authoring program may display a user interface of a (e.g., 3D) acoustic environment including a sound source. The authoring program may receive, via an input device, a user selection of a location within the UI toward which the sound source is to be oriented within the 3D acoustic environment. As an example, the sound source may be associated with an avatar of a user within the 3D acoustic environment.
[0110] The authoring program determines one or more orientation parameters that define the orientation of the sound source toward the location within the 3D acoustic environment (at box 162). In particular, based on the determined location, the authoring program can determine the orientation or viewing direction of the radiation pattern from the sound source to the location. In one aspect, the orientation can be relative to a global reference coordinate system (e.g., one or more axes) within the 3D environment. Based on the orientation, an orientation parameter such as a quaternion can be determined. The authoring program can store the one or more orientation parameters within the media program being authored by the content creator (e.g., its metadata).
[0111] In one aspect, the authoring program may perform these operations for each audio object or at least one audio object within the 3D acoustic environment so that individual orientation parameters may be determined. In one aspect, the authoring program may be configured to determine orientation parameters only for audio objects having a viewing direction that will be fixed at a specific point or position within the 3D environment.
[0112] The authoring program determines whether the position or sound source has moved within the 3D acoustic environment (at decision box 163). In particular, the viewing direction lock can determine whether an audio object within the 3D environment has moved relative to a reference point, or whether the point toward which the viewing direction of the audio object is locked has moved. The movement may occur within the media program after a certain period of time from when the viewing direction was locked. As described herein, the determination may be based on media metadata associated with the media program being authored and / or may be based on an analysis of the media program (such as determining whether the visual representation of the audio object has moved). As another example, when the point toward which the viewing direction is locked is an object (such as a virtual object within video content), the viewing direction lock may determine whether the virtual object has moved. As yet another example, when the position is the position of another audio object, the viewing direction lock may determine whether the audio object has moved based on the orientation data associated with the audio object.
[0113] If so, the authoring program updates the one or more orientation parameters to maintain the orientation of the sound source toward the location without user intervention and based on the movement (at block 164). In particular, the viewing direction lock can automatically update one or more of the orientation parameters without requiring the author of the audio content to manually adjust the viewing direction of the audio content through, for example, a UI of the authoring program. As an example, when the authoring program Figure 7 When the tour bus 143 moves from its position in the second stage 141 to the third stage 142 when viewing the 3D scene 152 of the video, the viewing direction lock 607 can automatically update the viewing direction, thereby alleviating the need for the content creator to manually update the viewing direction between the two stages. Returning to the previous example, the orientation can include the forward-facing direction of the user's avatar, and the radiation pattern can be the radiation pattern of the avatar's voice. In one aspect, to update the orientation parameters, the viewing direction lock can determine the position and orientation data of the sound source relative to the reference point before the movement and can determine the orientation data after the movement, and then update the orientation parameters based on the change in the orientation data. The viewing direction lock 607 can then store the updated orientation parameters to correspond to the movement at a specific time within the media program (e.g., with a timestamp).
[0114] In one aspect, the authoring program 606 may repeat at least some of the operations of process 160 for each scene within the media program. Thus, the authoring program fixes the orientation of the sound source toward the location (e.g., another audio object or a group of audio objects) in response to receiving user input, so that regardless of the motion of the location (e.g., movement of the object) or the motion of the sound source, the orientation of the radiation pattern of the sound source remains pointed toward the location. The orientation may be fixed during the duration of the media program, such as when the content creator is authoring the particular scene.
[0115] As described so far, the viewing direction can be fixed at a point within the 3D environment so that the radiation pattern of the audio object can always be oriented toward the point. On the other hand, the creation device can be configured to update the point to which the viewing direction is fixed based on an analysis of the media program, as described herein. As a useful illustration, when navigating social interactions in a large gathering of people, the gaze of a person can be directed. Upon arriving at the event, the focus can be directed toward the employee of the coat check, then it can be moved to a point on the horizon roughly in the direction of the stage where the live entertainment may be taking place, and then when the party guests move through the crowd, their focus can be directed toward a group of friends of colleagues who are also attending the event, and finally for the purpose of this example, the orientation can be transferred to individuals from a previously identified group that the party guests want to talk to. In one aspect, at various times during the scene, the creation device can determine whether the viewing direction should be adjusted. For example, the creation device can monitor the gaze of the party guests to determine when the gaze is transferred from the employee of the coat check to the horizon. When making this determination, the creation device can determine the position and update the orientation parameters accordingly.
[0116] Fig.10 Several stages 200 to 203 of a user interface (UI) 204 are shown, through which a user can select a point within a 3D scene, toward which an audio object remains fixed while movement occurs within the 3D scene. In particular, the UI 204 can be part of an authoring program 606 displayed on a display 605. Each of these stages shows a UI 204 including an audio object displayed on the display in a top-down view of the 3D scene. The UI also includes a listener position 208, which can be static within the 3D scene. In one aspect, the user interface can be an audio content authoring application that allows an author to create media content, as described herein.
[0117] The first stage 200 shows the creator selecting a point 207 within the UI 204 to which the viewing direction of the audio object 205 is to be fixed. In particular, this stage shows the viewing direction 206 toward the right hand side of the listener's position 208, so that if the listener at the position 208 is facing forward, the radiation pattern of the audio object will be perceived by the listener as originating from behind the listener and toward the right side of the listener. In one aspect, the viewing direction 206 may be predefined at this stage (e.g., based on metadata of the media program). In another aspect, the viewing direction may have been predefined (and fixed) by the user. In this case, the viewing direction 206 may point to the horizon.
[0118] In one aspect, authoring program 606 may display a UI of a 3D scene including an audio object as a sound source, wherein authoring program 606 may receive user input. In this case, UI 204 includes cursor 209, which may be controlled by an input device, such that a user may move cursor 209 to point 207, and user selection of a point within the UI by the input device may provide authoring program 606 with an indication that the location is where the audio object will be fixed. In another embodiment, the display may be a touch-sensitive display, wherein the user input may be a tap on the display at point 207.
[0119] The second stage 201 shows the user selecting the menu UI item 210 with the cursor 209 to lock the viewing direction 206 of the audio object 205 to the point 207. In particular, the authoring program 606 can receive the user selection of the menu UI item 210 through an input device (e.g., a mouse) that locks the orientation of the audio object to the point. The third stage 202 shows the result of the user selection of the menu UI item 210, which is a rotation of approximately 90° to the left and locking the viewing direction 206 to the point 207.
[0120] The first three stages show the 3D scene of the media program within UI 204 when the media program is at the start (start) time "[0:00:00]". The fourth stage 203 shows that one minute of the media program has passed, showing "[0:01:00]", and therefore the audio object 205 has moved from its initial position to a new position in front of and to the right of the listener position 208. This movement can be based on user input (e.g., the user selects and drags the audio object 205 from its original position to its new position), or can be performed automatically by the media program (e.g., based on the movement of the corresponding visual representation in the video content). When the audio object 205 has moved, the viewing direction 206 has remained fixed on the point 207.
[0121] Thus, as the audio object is translated to different locations in the coordinate system over time, the orientation parameter indicating the viewing direction of the audio object's radiation pattern may be continuously and automatically updated by the authoring program 606 to maintain the orientation toward that point. In one embodiment, as movement occurs, the updated orientation parameter may be stored in metadata associated with the media program to update the location / orientation of the audio object. In one aspect, the orientation parameter may be updated with a granularity that matches (or better than) the timing resolution of the system, such that as the object's location changes, the orientation parameter may be updated and stored in memory.
[0122] In one aspect, the specified point may remain stationary within the 3D scene, or move at different times during playback of the media program. In another aspect, the viewing direction may be set to various points within the 3D scene during at least a portion of the playback. For example, the viewing direction may be locked to a point for a certain period of time, which may be a static location, another audio object, multiple audio objects, etc. After the period of time, the point to which the viewing direction is locked may be switched to another point. For example, after another ten minutes in this example, the viewing direction 206 may be directed to another point within the UI 204. In one aspect, the different points may be set by the user, or may be automatically determined (selected) by the authoring program 606. For example, when an audio object is associated with an avatar of a user within the XR environment, the authoring program 606 may determine that the gaze of the avatar's eyes is focused from point 207 to another point.
[0123] As shown therein, the figure shows a user setting a point within a two-dimensional (2D) representation of the acoustic environment of a media program. On the other hand, UI 204 may show the acoustic environment in a 3D coordinate system (to show a 3D scene), and in this case, a user selection of a point within the 3D coordinate system may be selected.
[0124] Fig.11 is a block diagram of an audio system 100 according to one aspect, the audio system including a media content creation device 600 as an encoding side device (or encoding side), which can be configured to lock the viewing direction of the radiation pattern to a point within a 3D scene and provide metadata to a playback device as a decoding side to spatially render the media (sound) program. As described so far, the media content creation software program 606 can be configured to allow a content creator to create a media program and lock the viewing direction of the audio objects of the media program to a point during at least a portion of the playback of the media program. The playback can occur at the creation device, or it can occur at another device such as the playback device 133. In this case, the creation device can be configured to provide the media program and metadata for instructing the playback device how to spatially render the media program.
[0125] The authoring device 600 may be configured to encode the audio content 128 (e.g., one or more audio objects) and / or the optional video content 127 of the media program 608 into a bitstream 130 according to a codec such as the Moving Picture Experts Group (MPEG) standard, which may be transmitted to a playback device 133 via a network (e.g., the Internet). In one aspect, the authoring device may provide metadata 131 to the playback device (e.g., encoded within the bitstream 130 or provided separately), which may include orientation parameters 129 generated by the authoring software program and / or any other metadata 610 associated with the media program. In one aspect, the authoring device may provide metadata that may include orientation data, which may include the orientation of a sound source in a 3D space, and the metadata has one or more orientation parameters as an initial configuration of a 3D scene for playback by the playback device. Thus, the playback device may be configured to render one or more audio objects of the audio content according to the orientation parameters.
[0126] As described herein, other metadata 610 associated with the media program to be transmitted may be included in the bitstream (or transmitted separately) as metadata 131. In one aspect, the position of the audio object may be added to the metadata 131 as position data, which may be used by the playback device 133 to render the 3D scene spatially. For example, the position of the audio object may be a coordinate of a coordinate system, such as relative to an origin (such as Figure 7 The coordinate system may be a Cartesian coordinate system or a spherical coordinate system of a global scene origin 148). The orientation parameters may be added as rotation data indicating the rotation of the viewing direction of the radiation pattern projected from the audio object. The rotation data may include parameters such as yaw, pitch, and roll. When the coordinate system is a spherical coordinate system, the starting parameters may include spherical coordinates such as azimuth and elevation to indicate the viewing direction. In another embodiment, the parameters may include one or more quaternions as four normalized parameters.
[0127] In one aspect, the positions of audio objects and / or their corresponding (radiation pattern) orientations of a 3D scene may be provided by a playback device as metadata to indicate how the playback device spatially renders a media program of a 3D scene. For example, the positions of objects within a 3D scene may be encoded as position metadata including a scene tree structure that describes the relationship between the positions of audio objects and one or more origins within the 3D scene. Figure 7As shown, the scene may include a global scene origin 148 that may be the zero vector of a coordinate system (e.g., a Cartesian coordinate system, a spherical coordinate system, etc.) of the 3D scene, and may include one or more other origins (or sub-origins) associated with sub-scenes within the scene. For example, the tour bus 143 may be a sub-scene with a sub-origin at the tour bus position 149. In this case, metadata for the tour guide position 150a and the passenger position 150d may be encoded relative to the tour bus position 149. In addition, the tour bus position 149 and the pedestrian position 150b may be encoded relative to the global scene origin 148.
[0128] The orientation of the audio object can be added to the metadata 131 as one or more orientation parameters. In one aspect, the orientation of the audio object can include the orientation of the viewing direction of the corresponding radiation pattern derived from its position within the 3D scene, which can be relative to a coordinate system, as described herein. In one aspect, these orientations can take into account a locked viewing direction during the authoring of the media program. For example, during rendering and playback of the audio content, the orientation parameters of the radiation pattern of the audio object can cause the radiation pattern to follow or remain fixed on the object, as perceived at the listener's position.
[0129] In one aspect, the authoring device 600 may provide updates of at least some of the orientation data to the playback device 133. In particular, after providing the initial metadata, the authoring device may provide an updated set of parameters so that the playback device can adjust the orientation of the sound source during rendering. For example, upon determining that an audio object having a viewing direction pointing toward another audio object within a 3D scene has moved, the authoring device 600 may provide updated orientation parameters 129 as additional metadata. In one aspect, these updates may require less bandwidth than an initial configuration of the 3D scene, which may include orientation data for all (or at least some) audio objects within the 3D scene. On the other hand, when the media program is a live program that can be streamed to the playback device, the updates may be provided in real time.
[0130] The playback device 133 may be configured to receive the bitstream 130 and metadata 131 (which may be encoded in the bitstream), decode the audio content 128 and video content 127 of the media program, and may use the metadata 131 to render the audio content 128 spatially. In particular, the received metadata indicates how the playback device 133 renders the audio content 128 spatially, which may be based on the position of the listener. For example, the metadata may include a scene graph describing the position of audio objects within a 3D acoustic scene, and may include orientation parameters indicating the viewing direction of radiation parameters of one or more of those audio objects. The spatial audio renderer 101 may be configured to render the audio content spatially according to the metadata about the listener's position to generate one or more audio signals including audio content for playback through one or more speakers 123. The resulting sound output provides the listener with a perception of being within the 3D scene and experiencing a sound source about the listener's position. In one aspect, the listener's position may be the listener's position within the 3D scene. For example, where the 3D scene of the media program is an XR environment that the listener (e.g., his avatar) is participating in, the listener's position may be where the avatar is located within the XR environment.
[0131] In one aspect, playback device 133 may (e.g., periodically) receive updated metadata 131 from authoring device 600 that may indicate updated positional data and / or updated orientation parameters for one or more objects within the 3D scene. Spatial audio renderer 101 may use the updated orientation parameters to redirect the viewing direction of the radiation pattern based on movement within the 3D so that a listener may perceive the radiation pattern to remain directed toward a particular point within the 3D scene.
[0132] In one aspect, when a listener is viewing a 3D scene (such as Figure 7 When viewing a 3D scene 152 of a video image, the listener may be located at a viewing position within the global scene, such as near pedestrian position 150b. The spatial renderer 101 may be configured to render the audio content 128 using the position data received within the metadata 131 so that the sound may be perceived as consistent with the video content 127, which may show the tour bus 143 and pedestrians 147. Therefore, it is important to use dynamic updating of the position data of the sound sources as movement occurs within the 3D scene 152. By locking the viewing direction for at least some audio objects, the orientation parameters of those objects may be automatically updated to account for the displacement of their coordinates relative to the position where the viewing direction is fixed. This allows the content creator to maintain the desired directivity of the radiation pattern of any audio object without having to manually update the corresponding orientation parameters of objects (such as objects that move over time within the 3D scene).
[0133] In one aspect, optional video content 127 may be decoded and played back via display 122. In one aspect, video playback may be synchronized with audio content playback.
[0134] In one aspect, the system 100 may perform a viewing direction locking operation of a media program in real time. In this case, the operation described herein may be performed continuously (or periodically) on the stream of the audio content of the media program. For example, the operation of determining the point to which the viewing direction will be locked can be determined from the beginning (or before) of the media program (or determined at the start time when the media program will be streamed). Once streaming begins, when movement occurs within the scene of the streamed media program, the creation device may adjust the orientation parameters of the viewing direction so as to maintain the viewing direction of the audio object, as described herein. In one aspect, the creation device may continue to update the orientation parameters until the end of the media program (or the stop time when the media program is no longer streamed in real time). In some aspects, the creation device 600 may periodically update or change the point at which the viewing direction is locked based on various criteria, as described herein. For example, the creator of the media program may adjust the point at which the viewing direction is locked between the scenes of the media program. In one aspect, as described herein, the creation device may be configured to play back the content of the streamed content, or may be configured to encode and transmit the content for decoding and playback by the playback device 133.
[0135] Thus, operations may be performed during a live broadcast of a media program, such as a live audio communication (e.g., a live video conference call, an audio call, etc.) or a live football game. For this example, the creation device 600 may be configured to receive a sound program as a live audio (and / or video) broadcast, and may be configured to determine the acoustic characteristics of audio objects within the live broadcast (such as referees and players within the game). When determining the audio objects, characteristics such as viewing direction, the position of the audio objects within the scene, and the radiation pattern may be determined. The creation device may be configured to lock one or more viewing directions, as described herein. For the football game example, the creation device may lock the viewing direction of the referee in a direction away from the sidelines and toward the direction the referee is facing. Thus, when the soundscape is spatially rendered at the creation device or playback device, the listener may perceive the referee speaking in the distance when the listener's position is at the sidelines.
[0136] In the case of live audio communication, the authoring device can be configured to capture audio using one or more microphones 302 and / or capture video using camera 301, and can (encode and) transmit the audio / video to a playback device. In doing so, the authoring device can be configured to determine one or more sound sources within the audio, and can be configured to determine the radiation pattern and / or corresponding viewing direction based on analysis of the audio and / or video or based on user input, as described herein. In this case, the playback device can use the metadata generated during the live audio communication to spatially render the audio content.
[0137] As described thus far, the authoring device may be configured to lock the viewing direction by updating one or more orientation parameters of the audio object so that its orientation remains focused on a point or location within the 3D scene during playback of the audio content. In particular, the authoring device updates the parameters and may store those parameters with the media program and / or may provide the updated parameters to the playback device, which may then apply the updated parameters during spatial rendering of the audio content. These updated parameters may be received in metadata as part of the orientation data indicating the position and orientation of the audio object. Therefore, the operations described herein may be useful in application areas such as content creation and content encoding, transmission, and spatial rendering.
[0138] As described herein, the bitstream 130 may already include audio object metadata, wherein each audio object may be identified by a playback device, for example, by using a unique identifier (such as an enumeration), and wherein the object metadata may include an object position. For example, the position of the audio object may be included in the orientation data having the position of the audio object relative to an origin within the 3D scene, and may include the orientation of the audio object as one or more orientation parameters.
[0139] In one aspect, view direction lock metadata may be preserved during audio encoding and rendering. The metadata may be provided by an authoring device to a playback device for spatial rendering, wherein the metadata may indicate a location within a 3D scene toward which a view direction of a radiation pattern of an audio object will remain oriented as the location or audio object moves during spatial rendering. In one aspect, the view direction lock metadata may be included within the metadata 131 or may be provided separately to the playback device. Providing an efficient description of the radiation pattern orientation may reduce the bit rate at which metadata is encoded and transmitted because multiple radiation patterns may reference the same location within the 3D scene specified by the view direction lock metadata without the need to transmit separate orientation parameters for each radiation pattern.
[0140] Fig.121 is a flow chart of one aspect of a process 170 for encoding metadata indicating a location within a 3D scene toward which a radiation pattern of an audio object remains oriented during playback of the sound program at an authoring device 600. The process 170 begins by receiving a sound program including an audio object within a 3D scene (at block 171). The authoring device 600 encodes the sound program into a bitstream 130 (at block 172).
[0141] The authoring device 600 encodes metadata for the sound program into the bitstream, which metadata may include viewing direction locking metadata that indicates a position within the 3D scene toward which the radiation pattern of the audio object will remain oriented when the position or audio object moves within the 3D scene during spatial audio rendering of the sound program (at box 173). In particular, the authoring device may generate viewing direction locking metadata, which is a structure with a payload indicating that the viewing direction will be fixed to the position of the playback device, which may be based on user input and / or analysis of the sound program, as described herein. In one aspect, the viewing direction locking payload within the metadata may be based on a position or point to which the radiation pattern will be fixed. For example, when the position is a static position within the 3D scene, the payload may include orientation data, such as coordinates of a coordinate system, or may include a reference to an object within the 3D scene (e.g., a unique identifier). More content about the content of viewing direction locking metadata is described herein.
[0142] The authoring device 600 encodes audio object metadata that includes a reference to view direction lock metadata that instructs the playback device to lock the direction of the audio object's radiation pattern to the position indicated by the view direction lock metadata during spatial audio rendering of the sound program (at box 174). Specifically, the authoring device can determine a point within the 3D scene at which the view direction of the audio object's radiation pattern is to be fixed, and when making this determination, it can determine which view direction lock metadata includes the particular point, and then add a reference to the view direction lock metadata to the audio object metadata. In one aspect, the audio object metadata may include other metadata described herein, such as the corresponding radiation pattern, the position of the audio object within the 3D scene, etc.
[0143] In one aspect, at least some of the operations described herein may be optional and / or may be performed in combination with other operations described herein. For example, the operations of block 171 and / or block 172 may be optional.
[0144] Figures 13a to 13cMetadata that includes a radiation pattern that indicates how a playback device is to direct an audio object during playback of a sound program. Each of these figures shows a different example of a structure or payload of viewing direction lock metadata that indicates a position at which a viewing direction is to be fixed as the audio object moves during playback by the playback device.
[0145] Fig.13a An example of viewing direction locking metadata for fixing the viewing direction at a static position within a 3D scene is shown. In one aspect, the static position may be a point within the 3D scene that may not move relative to a reference point (such as a global origin). In another aspect, it may be an object, such as a pebble, a position on the ground, a point on the horizon, etc. The figure shows audio object A metadata 190a and viewing direction locking metadata 194. The audio object metadata includes a payload having a radiation pattern 191a, a viewing direction locking reference 192a, and an object position 193a. In one aspect, the object position 193a may include at least a portion of the orientation data described herein. For example, the orientation data may include a scene graph describing the position of the audio object A relative to a reference point of the 3D scene. In some aspects, the audio object metadata may include other information, such as reverberation width, pre-delay, etc. The radiation pattern 191a may include an index selected from a dictionary of radiation patterns, which may be used by the decoding side to select a corresponding radiation pattern from a codebook and use the pattern during spatial rendering. The viewing direction locking reference 192a may include a unique identifier that may be associated with the viewing direction locking metadata 194. In one aspect, the unique identifier may be generated by the encoding side when generating the viewing direction lock metadata.
[0146] The viewing direction lock metadata 194 includes a viewing direction position 195, which may be a position within the 3D scene. For example, the position may be a set of coordinates of the position within a coordinate system of the 3D scene. In one aspect, the position may be coordinates relative to a reference point within the 3D scene, such as the origin of the 3D scene (composed of a scene graph, as described herein).
[0147] Fig.13bAn example of viewing direction locking metadata for fixing a viewing direction on another audio object (such as an audio object) within a 3D scene is shown. As described herein, the authoring device 600 can provide metadata including positional data of audio objects of a sound program. In addition to providing metadata for audio object A, the authoring device 600 also provides audio object B metadata 190b that includes (at least) the position 193b of object B. Thus, the viewing direction locking metadata 194 includes a reference (e.g., a unique identifier) 196 of audio object B, where the radiation pattern 191a of audio object A is to be directed within the 3D scene. As described herein, the playback device can determine that the location of the viewing direction is the object position 193b of audio object B, and therefore direct the radiation pattern 191a of audio object A toward the object position 193b during playback. In addition, the playback device can ensure that the radiation pattern remains directed toward audio object B when either audio object moves within the 3D scene during spatial audio rendering of the sound program. Thus, the metadata 194 uses the object identifier as a proxy for the viewing direction point.
[0148] Fig.13c An example of viewing direction locking metadata for fixing the viewing direction at a point based on multiple objects within a 3D scene is shown. As described herein, the viewing direction can be locked to a point within an area surrounded by an audio object. In particular, the authoring device 600 can be configured to determine the point and then provide the metadata to the playback device. In this figure, the authoring device can provide a set of references to audio objects at different locations within the 3D scene so that the playback device can determine the point to which the viewing direction is to be locked. Specifically, the viewing direction locking metadata 194 includes references 196 to several audio objects (i.e., object B metadata 190b including the position 193b of audio object B to object N 190n including the position 193n of the object). In one aspect, each of objects B to N can be determined based on user input and / or analysis of the media program, as described herein. On the other hand, the object can be determined based on a user selection of a point surrounded by an area defined by the object. The playback device can be configured to determine the point to which the viewing direction points based on the area surrounded by the object positions 193b to 193n, as described herein (e.g., to the centroid).
[0149] In one aspect, the authoring device 600 may be configured to update at least some of the metadata and provide the updated metadata to the playback device. Fig.13aIn the case of , if the look direction position 195 is associated with a moving object (e.g., a car), upon determining that the car has moved, the authoring device may adjust the look direction position 195 and provide updated metadata to the playback device. In contrast, when the look direction locked metadata includes references to audio objects, the authoring device may not need to retransmit or update the look direction metadata when those objects move. For example, in Fig.13b In the case of audio object B being moved, the authoring device may be configured to update the object position 193b and provide the updated metadata to the playback device. On the other hand, the reference to the viewing direction lock metadata may be adjusted based on a change in the viewing direction of the object, which may occur during playback of the media program.
[0150] As described herein, an authoring device may provide metadata including a location at which a playback device directs a radiation pattern of an audio object during spatial rendering. In one aspect, metadata 194 may include one or more angular offsets (e.g., in quaternions or angular parameters such as azimuth and elevation) that, when applied by a playback device, may cause the viewing direction of the radiation pattern to remain offset from the location referenced by metadata 194. Thus, as the location indicated by the viewing direction lock metadata or the audio object moves within a 3D scene, the radiation pattern may remain offset from that location. In the case where the audio object is a sound source for a first avatar of a first user within an XR environment and the location is a second avatar of a second user, the offset may ensure that the radiation pattern is directed upward and toward the face of the second avatar rather than the body of the second avatar (which may be where the location was selected). Thus, the offset provides some objects with one or more different 3D rotations that may be useful in terms of facing a point.
[0151] Fig.14180 is a flowchart of one aspect of a process 180 for decoding metadata at a playback device 133 that instructs a decoding side to direct a radiation pattern of an audio object toward a point within a 3D scene during playback of a sound program. Process 180 begins with the playback device receiving a bitstream that includes encoded audio objects for a sound program and / or associated metadata (at box 181). In one aspect, the associated metadata may include audio object metadata that instructs the decoding side how to spatially render the audio object, as described herein. The playback device decodes the audio object for the sound program from the bitstream (at box 182). The playback device receives metadata for the sound program that instructs the decoding side to lock a viewing direction of a radiation pattern of the audio object toward a point within the 3D scene of the sound program or the audio object as the point moves during playback of the sound program (at box 183). In one aspect, the metadata may be received within the bitstream and decoded from the bitstream, or may be received separately. The metadata may include a viewing direction locking metadata structure and an audio object metadata structure, such as Figures 13a to 13c In this case, the audio object metadata structure may include a reference to the viewing direction metadata, wherein the decoding side may be configured to lock the viewing direction of the radiation pattern of the audio object in response to receiving the audio object metadata.
[0152] The playback device 133 determines whether the received metadata references a group of audio objects (at decision block 184). Specifically, the playback device may determine whether the viewing direction lock metadata 194 includes references 196 to two or more objects within the 3D scene, such as Fig.13c As shown. If so, the playback device determines the position of the point that locks the viewing direction based on the position of the group of audio objects in the 3D scene (at box 186). In particular, the playback device can be configured to determine the position of the object based on the reference in the viewing direction locking metadata. The playback device can determine the position of the point in the 3D scene based on these positions. For example, the playback device can identify an area within the 3D scene that can be defined by the position of the referenced object, and can select the position within the area. In one aspect, the position within the area can be the centroid of the identified area. In one aspect, the referenced object can be an audio object (sound source) within the 3D scene, or it can be a visual object within the 3D scene, or a combination thereof. However, if the metadata does not reference a group of objects, the playback device determines the position of the point within the 3D scene based on the metadata (at box 185). In particular, the playback device can determine the position based on the coordinates in the viewing direction locking metadata, or it can determine the position as the position of the audio object based on the viewing direction locking metadata that includes a reference to another audio object.
[0153] The playback device 133 determines a viewing direction of the radiation pattern of the audio object from the position of the audio object to the position of the point within the 3D scene (e.g., as one or more orientation parameters) (at block 187). In particular, the playback device may determine the one or more orientation parameters based on the received orientation data of the audio object, the one or more orientation parameters defining the direction (e.g., yaw, pitch, and roll) from the position of the audio object to the position of the point relative to a coordinate system, the positions being relative to a reference point, the orientation data of the audio object may include a scene graph including an origin and the position of the audio object.
[0154] The playback device 133 determines the listener position (at box 188). For example, the listener position can be determined based on sensor data captured by one or more sensors, such as an inertial measurement unit of an electronic device (e.g., a head-mounted device) worn by the listener. On the other hand, the listener position can be fixed within the 3D scene, or can be movable within the 3D scene. The listener position can be determined relative to a reference point within the 3D scene, such as a global origin (to which the position of the audio object can also be referenced). The playback device spatially renders the audio object based on the listener position and the viewing direction of the radiation pattern (at box 189). In particular, the playback device can use one or more orientation parameters associated with the viewing direction and spatially render a sound program including the audio object according to the listener position to generate one or more spatially rendered audio signals that can be used to drive one or more speakers.
[0155] In one aspect, for audio content that allows the listener position to change within a 3D scene (such as six degrees of freedom content), the listener position may move through the 3D scene. In this case, some radiation patterns may be intended to always face the listener. In this case, the viewing direction lock reference of the audio object may reference the listener position, and the playback device may be configured to calculate the viewing direction of the radiation pattern orientation based on the current listener position in the 3D scene, and may be configured to render the sound source spatially according to the listener position (orientation parameters associated with the listener position).
[0156] The playback device 133 determines whether the point and / or audio object has moved within the 3D scene (at decision block 700). For example, the playback device may receive updated orientation data from the authoring device indicating that the position of the audio object has moved from its previous position to a new position within the 3D scene. This may be the case when the audio object is associated with a speaking person who walks from one place in a room to another. If so, the playback device updates the viewing direction to maintain a radiation pattern pointing to the point (at block 701). To do this, the playback device may adjust at least one of the orientation parameters to account for the change in the orientation of the audio object. Thus, as the point changes dynamically, the position metadata (e.g., position 195 in viewing direction lock metadata 194 or object position 193b in audio object B metadata 190b, for example, as shown in FIG. 704 ) may be updated. Fig.13a and Fig.13b ) may be updated accordingly in the bitstream received by the playback device.
[0157] In one aspect, the playback device can adjust the viewing direction within a certain period of time. As an example, the playback device can receive additional metadata indicating that the audio object B has moved from the original position to which its radiation pattern is locked to a new position within the 3D scene. In order to avoid audible artifacts when the position of the viewing direction receives a sudden large change (for example, the audio object B moves quickly from its original position to a new position within a short period of time), the playback device can move the viewing direction of the radiation pattern from the original position of the audio object B to its new position within a certain period of time. For example, the decoding side can determine the angle (or one or more angles) between the original position of the audio object B and its new position relative to the position of the audio object A within the 3D scene. The playback device can move the viewing direction by rotating the viewing direction along the angle around the position of the audio object at a speed based on the time period. Therefore, the playback device can generate different orientation parameters at given intervals of the time period along the angle, and can use the newly generated parameters to render the sound program in space. In one aspect, the playback device can interpolate between the current position and the updated position to modify the viewing direction of the radiation pattern with multiple small angle increments along the smooth trajectory of the angle. On the other hand, the interpolation may also be applicable when an audio object having a radiation pattern is moved relative to a position towards which the viewing direction of its radiation pattern is locked. Thus, by having the playback device interpolate between two positions, the bitrate of the bitstream may be reduced while providing a smooth trajectory between the two positions.
[0158] The playback device may limit movement of the viewing direction when the point toward which the viewing direction is directed changes. For example, the viewing direction lock metadata may be updated to reference the new point. This may be the case when the viewing direction of a sound source changes, such as from pointing toward an employee at a cloakroom to a horizontal plane. Thus, the playback device may receive updated viewing direction lock metadata and, in response to receiving the updated payload, may move the viewing direction from its original point to the new point for a period of time, as described herein, in order to limit audible artifacts during playback.
[0159] Fig.15 A block diagram of audio processing system hardware (e.g., audio processing system hardware of the authoring device 600 and / or playback device 133 of the system 100) that can be used with any of the aspects described herein in one aspect is shown. The audio processing system can represent a general purpose computer system or a special purpose computer system. Note that although Fig.15 Various components of an audio processing system are shown that may be incorporated into one or more of the devices described herein, but this is merely one example of a particular implementation and is merely to illustrate the types of components that may be present in the system. Fig.15 It is not intended to represent any particular architecture or manner of interconnecting these components, as such details are not germane to the various aspects described herein. It should also be understood that a circuit having a greater Fig.15 Other types of audio processing systems may include fewer or more components than those shown. Therefore, the processes described herein are not limited to those described herein. Fig.15 Used together with hardware and software.
[0160] like Fig.15 As shown, an audio processing system (or system) 300 (e.g., a laptop, desktop computer, mobile phone, smart phone, tablet computer, smart speaker, head mounted display (HMD), headset (headphone), or infotainment system for an automobile or other vehicle) includes one or more buses 308 for interconnecting various components of the system. As is known in the art, one or more processors 307 are coupled to the bus 308. The one or more processors may be a microprocessor or a special purpose processor, a system on a chip (SOC), a central processing unit, a graphics processing unit, a processor created by an application specific integrated circuit (ASIC), or a combination thereof. The memory 306 may include a read-only memory (ROM), a volatile memory, and a non-volatile memory, or a combination thereof, coupled to the bus using techniques known in the art. The camera 301, the microphone 302, the speaker 123, and the display 122 may be coupled to the bus.
[0161] Memory 306 may be connected to the bus and may include DRAM, a hard drive or flash memory, or a magnetic optical drive or magnetic storage, or an optical drive or other type of memory system that maintains data even after the system is powered off. In one aspect, processor 307 retrieves computer program instructions stored in a machine-readable storage medium (memory) and executes these instructions to perform the operations described herein.
[0162] Although not shown, the audio hardware may be coupled to one or more buses 308 to receive audio signals to be processed and output by the speaker 123. The audio hardware may include a digital-to-analog converter and / or an analog-to-digital converter. The audio hardware may also include an audio amplifier and a filter. The audio hardware may also be connected to a microphone 302 (e.g., a microphone array) to receive audio signals (whether analog or digital) and transmit the signals to the bus 308.
[0163] The network interface 305 can communicate with one or more remote devices and networks. For example, the interface can communicate via known technologies such as Wi-Fi, 3G, 4G, 5G, Bluetooth, ZigBee, or other equivalent technologies. The interface may include a wired or wireless transmitter and receiver that can communicate (e.g., receive and transmit data) with networked devices such as a server (e.g., the cloud) and / or other devices such as a remote speaker and a remote microphone.
[0164] It should be understood that the aspects disclosed herein may utilize memory away from the system, such as a network storage device coupled to the audio processing system via a network interface such as a modem or an Ethernet interface. As is well known in the art, the bus 308 can be connected to each other via various bridges, controllers, and / or adapters. In one aspect, one or more network devices may be coupled to the bus 308. One or more network devices may be wired network devices (e.g., Ethernet) or wireless network devices (e.g., WI-FI, Bluetooth). In some aspects, the various aspects described may be performed by a networked server that communicates with one or more devices.
[0165] Various aspects described herein may be embodied at least in part in software. That is, these techniques may be implemented in an audio processing system in response to its processor executing instructions contained in a storage medium such as a non-transitory machine-readable storage medium (e.g., dynamic random access memory, static memory, non-volatile memory). It should be noted that the phrase "processor" is generally used herein to refer to one or more processors, which may be in separate housings or devices and may communicate with each other, for example, to actually form a distributed computing system. In addition, in various aspects, hard-wired circuits may be used in combination with software instructions to implement the techniques described herein. Therefore, these techniques are not limited to any specified combination of hardware circuitry and software, nor to any source of instructions executed by the audio processing system.
[0166] In this specification, certain terms are used to describe the features of various aspects. For example, in some cases, the terms "module", "processor", "unit", "renderer", "system", "device", "filter", "engine", "block", "detector", "simulation", "model", "identifier", "lock" and "component" represent hardware and / or software configured to perform one or more processes or functions. For example, examples of "hardware" include, but are not limited to, integrated circuits such as processors (e.g., digital signal processors, microprocessors, application specific integrated circuits, microcontrollers, etc.). Therefore, as understood by those skilled in the art, different combinations of hardware and / or software can be implemented to perform the processes or functions described by the above terms. Of course, the hardware can alternatively be implemented as a finite state machine or even a combinational logic component. Examples of "software" include executable code in the form of applications, applets, routines, and even a series of instructions. As described above, the software can be stored in any type of machine-readable medium.
[0167] Certain portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits in a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of audio processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein and is contemplated to refer to a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. However, it should be kept in mind that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, it will be apparent from the foregoing discussion that it will be understood that throughout this specification, discussions using terms such as those given in the claims below refer to actions and processes of an audio processing system or similar electronic device that manipulates data represented as physical (electronic) quantities in the system's registers and memories and converts them into other data similarly represented as physical quantities in the system's memories or registers or other such information storage, transmission or display devices.
[0168] The processes and blocks described herein are not limited to the specific examples described, and are not limited to the specific order used as examples in this article. On the contrary, any processing box can be reordered, combined or removed, executed in parallel or serially as needed to achieve the above results. The processing blocks associated with the implementation of the audio processing system can be executed by one or more programmable processors to execute one or more computer programs stored on a non-transient computer-readable storage medium to perform the functions of the system. All or part of the audio processing system can be implemented as a dedicated logic circuit (e.g., FPGA (field programmable gate array) and / or ASIC (application-specific integrated circuit)). All or part of the audio system can be implemented using an electronic hardware circuit including at least one of an electronic device such as a processor, a memory, a programmable logic device or a logic gate. In addition, the process can be implemented in any combination of hardware devices and software components.
[0169] In some aspects, the present disclosure may include language such as “at least one of [element A] and [element B]”. The language may refer to one or more of these elements. For example, “at least one of A and B” may refer to “A”, “B”, or “A and B”. Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B” or “at least either A or B”. In some aspects, the present disclosure may include language such as “[element A], [element B], and / or [element C]”. The language may refer to any one of these elements or any combination thereof. For example, “A, B, and / or C” may refer to “A”, “B”, “C”, “A and B”, “A and C”, “B and C”, or “A, B, and C”.
[0170] While certain aspects have been described and shown in the drawings, it is to be understood that such aspects are illustrative only and not restrictive, and that the disclosure is not limited to the particular construction and arrangements shown and described, since various other modifications may occur to those skilled in the art.
[0171] To assist the Patent Office and any reader of any patent issuing in this application in interpreting the appended claims, Applicants wish to note that they do not intend for any of the appended claims or claim elements to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are expressly used in a claim.
[0172] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are deemed to meet or exceed industry or government requirements for maintaining user privacy. Personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly stated to the user.
Claims
1. A decoding-side method for spatial audio rendering using metadata, the method include: Decoding a plurality of audio objects, a plurality of channels or a HOA representation of a sound program from a bitstream; as well as receiving metadata of the sound program, wherein the metadata comprises: A first message indicating whether decoding-side processing during playback of the sound program is to apply scene reverberation: i) on a per-audio object basis or on a group of multiple objects from the multiple audio objects; ii) on a per-channel basis or on a group of multiple channels from the multiple channels; or iii) based on the HOA representation.
2. The method of claim 1, wherein the metadata further comprises an index of an impulse response (IR) selected from a plurality of IRs, the method further comprising: include: not applying scene reverberation during the playback according to the instructions in the first message; or The scene reverb is applied during the playback according to the index specified in the first message.
3. The method according to claim 1, wherein the metadata further comprises an index to a set of reverberation parameters, the method further comprising: include: In response to the first message, not applying scene reverberation during the playback; applying the scene reverberation during the playback according to the index to the set of reverberation parameters specified in the first message; or The scene reverb is applied during the playback according to the IR modified by the reverb parameters specified in the first message.
4. The method of claim 1, wherein the metadata comprises the complete room geometry or an index to a selected set of reverberation parameters based on the room geometry, the method further comprising: include: According to the first message, no scene reverberation is applied during the playback; applying the scene reverberation during the playback according to the complete room geometry as indicated by the first message; or As indicated by the first message, the scene reverberation is applied during the playback according to a selected set of room geometry based reverberation parameters.
5. The method according to claim 1, wherein the metadata further comprises a second message, the method further comprising: include: not applying post-processing reverberation after the scene reverberation during the playback according to the second message; or As indicated by the second message, the post-processing reverb is applied according to a default acoustic environment after the scene reverb during the playback.
6. The method according to claim 5, further comprising: include: not applying post-processing reverberation as indicated by the second message; applying the post-processing reverberation according to the default acoustic environment as indicated by the second message; or The post-processing reverb is applied based on a shortened IR or an early reflections IR only, as indicated by the second message.
7. The method of claim 1, wherein the metadata includes reverberation parameters such as reverberation parameters based on room geometry, the method further comprising: include: not applying scene reverberation according to the first message; or The scene reverberation is applied according to the reverberation parameters.
8. The method of claim 1, wherein the metadata comprises a set of filter coefficients, the method further comprising: include: not applying scene reverberation according to the first message; or The scene reverberation is applied according to a reverberation digital filter defined by the set of filter coefficients.
9. A method for spatial audio rendering using metadata, performed by a decoding side, the method include: Decodes the sound program's audio object from the bitstream; as well as receiving metadata of the sound program, wherein the metadata comprises: A first message instructing the decoding side as to which radiation pattern of a plurality of radiation patterns stored at the decoding side to apply to the audio object during spatial audio rendering for playback of the sound program. 10 . The method of claim 9 , wherein the first message comprises a first index representing a type of the audio object and a second index representing a viewing direction or orientation of the audio object.
11. The method according to claim 10, further comprising: include: identifying a selected one of the radiation patterns based on the first index and the second index; as well as The audio object is spatially audio rendered using the selected one of the radiation patterns.
12. The method of claim 9, further comprising obtaining a position of a listener during the playback, wherein the spatial audio rendering is further based on the position of the listener.
13. The method according to claim 9, further comprising: include: decoding another audio object of the sound program from the bitstream, The metadata further includes instructions to the decoding side to apply one of the following items contained in the metadata to the other audio object during playback spatial audio rendering to generate the audio object with a radiation pattern: a set of digital filter coefficients defining the impulse response; A set of HOA coefficients based on each frequency band; or Based on a set of radiation patterns for each frequency band.
14. The method of claim 9, wherein each of the radiation patterns comprises a shape of omni, cardioid, hypercardioid, or dipole.
15. An electronic device, the electronic device include: at least one processor; and a memory having instructions stored therein that, when executed by the at least one processor, cause the electronic device to: Decoding a plurality of audio objects, a plurality of channels or a HOA representation of a sound program from a bitstream; as well as receiving metadata of the sound program, wherein the metadata comprises: A first message indicating whether decoding-side processing during playback of the sound program is to apply scene reverberation: i) on a per-audio object basis or on a group of multiple objects from the multiple audio objects; ii) on a per-channel basis or on a group of multiple channels from the multiple channels; or iii) based on the HOA representation.
16. The electronic device of claim 15, wherein the metadata further comprises an index of an impulse response (IR) selected from a plurality of IRs, wherein the memory has further instructions for: According to the instructions in the first message, no scene reverberation is applied during the playback; or The scene reverb is applied during the playback according to the index specified in the first message.
17. The electronic device of claim 15, wherein the metadata further comprises an index to a set of reverberation parameters, the memory having further instructions for: In response to the first message, not applying scene reverberation during the playback; applying the scene reverberation during the playback according to an index to the set of reverberation parameters specified in the first message; or The scene reverb is applied during the playback according to the IR modified by the reverb parameters specified in the first message.
18. The electronic device of claim 15, wherein the metadata comprises a complete room geometry or an index to a selected set of room geometry-based reverberation parameters, the memory having further instructions for: According to the first message, no scene reverberation is applied during the playback; applying the scene reverberation during the playback according to the complete room geometry as indicated by the first message; or As indicated by the first message, the scene reverberation is applied during the playback according to a selected set of room geometry based reverberation parameters.
19. The electronic device of claim 15, wherein the metadata further comprises a second message, the memory having further instructions for: According to the second message, no post-processing reverb is applied after the scene reverb during the playback; or As indicated by the second message, the post-processing reverb is applied according to a default acoustic environment after the scene reverb during the playback.
20. The electronic device of claim 19, wherein the memory has further instructions for: not applying post-processing reverberation as indicated by the second message; applying the post-processing reverberation according to the default acoustic environment as indicated by the second message; or The post-processing reverb is applied based on a shortened IR or an early reflections IR only, as indicated by the second message.
21. The electronic device of claim 15, wherein the metadata includes reverberation parameters such as reverberation parameters based on room geometry, the memory having further instructions for: According to the first message, scene reverberation is not applied; or The scene reverberation is applied according to the reverberation parameters.
22. An encoding-side method for spatial audio rendering using metadata, the method include: Encoding multiple audio objects, multiple channels or HOA representations of a sound program into a bitstream; as well as Providing metadata of the sound program, wherein the metadata includes: A first message indicating whether decoding-side processing during playback of the sound program is to apply scene reverberation: i) on a per-audio object basis or on a group of multiple objects from the multiple audio objects; ii) on a per-channel basis or on a group of multiple channels from the multiple channels; or iii) based on the HOA representation.
23. The method of claim 22, wherein the metadata further comprises an index of an impulse response (IR) selected from a plurality of IRs, and the first message indicates that the decoding side processes: Do not apply scene reverb; or The scene reverb is applied according to the index.
24. The method of claim 22, wherein the metadata further comprises an index to a set of reverberation parameters, and the first message instructs the decoding side to process: Do not apply scene reverb; applying the scene reverberation according to the index to the set of reverberation parameters; or The scene reverb is applied according to the IR modified by the reverb parameters.
25. The method of claim 22, wherein the metadata comprises a complete room geometry or an index to a selected set of room geometry-based reverberation parameters, and the first message instructs the decoding side to process: Do not apply scene reverb; or applying the scene reverberation according to the complete room geometry; The scene reverberation is applied according to a selected set of room geometry based reverberation parameters.
26. The method of claim 22, wherein the metadata further comprises a second message indicating whether the decode-side processing applies post-processing reverberation during the playback to: No post-processing reverb is applied; or The post-processing reverb is applied according to a default acoustic environment.
27. The method according to claim 26, wherein the second message indicates that the decoding side processing is: No post-processing reverb is applied; applying the post-processing reverberation according to the default acoustic environment; or The post-processing reverb is applied based on the shortened IR or the early reflections IR only.
28. The method of claim 22, wherein the metadata includes reverberation parameters such as reverberation parameters based on room geometry, and the first message instructs the decoding side to process: Do not apply scene reverb; or The scene reverberation is applied according to the reverberation parameters.
29. The method of claim 22, wherein the metadata comprises a set of filter coefficients and the first message instructs the decoding side to process: Do not apply scene reverb; or The scene reverberation is applied according to a reverberation digital filter defined by the set of filter coefficients.
30. An encoding-side method for spatial audio rendering using metadata, the method include: Encode the audio objects of the sound program into a bitstream; as well as Providing metadata of the sound program, wherein the metadata includes: A first index and a second index to a lookup table of multiple radiation patterns stored in a decoding side, wherein the first index and the second index together point to a selected one of the multiple radiation patterns, and the metadata indicates that the decoding side applies the selected one of the multiple radiation patterns to the audio object during playback spatial audio rendering of the sound program.
31. The method of claim 30, wherein the first index indicates a type of object and the second index indicates a direction in which a major axis of an object of the type is located.
32. The method of claim 30, wherein the second index refers to a yaw, pitch and roll triplet or quaternion.
33. The method according to claim 30, further comprising: include: encoding another audio object of the sound program into the bitstream, wherein the another object is of a type not in the lookup table; as well as providing one of the following in the metadata to the decoding side together with instructions to apply the one of the following to the other audio object during the playback spatial audio rendering: a set of digital filter coefficients defining the impulse response; A set of HOA coefficients per frequency band; or Based on a set of radiation patterns for each frequency band.
34. The method of claim 33, wherein each of the radiation patterns comprises an omnidirectional, cardioid, supercardioid, or dipole shape.
35. A method, the method comprising: include: a user interface (UI) for displaying a three-dimensional (3D) acoustic environment including sound sources; receiving, via an input device, a user selection of a location within the UI toward which the sound source is to be directed within the 3D acoustic environment; determining a set of parameters defining an orientation of the sound source toward the location within the 3D acoustic environment; determining a motion of the location or the sound source within the 3D acoustic environment; as well as The set of parameters is updated, without user intervention and based on the movement, to maintain the orientation of the sound source toward the location.
36. A method according to claim 35, wherein the user selection is a first user selection, wherein the method further comprises receiving a second user selection of a UI item through the input device, the second user selection of the UI item locking the orientation and the position of the sound source, wherein the set of parameters is determined in response to receiving the second user selection.
37. The method of claim 35, wherein the sound source is a first sound source and the location is one of: a static position within the 3D acoustic environment; a second sound source within the 3D acoustic environment; or A point within an area surrounded by a plurality of sound sources within the 3D acoustic environment.
38. The method of claim 37, wherein the motion of the position comprises movement of the second sound source from the position to a new position within the 3D acoustic environment, wherein the set of parameters is updated such that the orientation of the first sound source follows the second sound source from the position to the new position.
39. The method of claim 35, wherein the sound source comprises a radiation pattern of sound from the sound source, wherein the orientation is a viewing direction of the radiation pattern from the sound source and towards the location.
40. The method of claim 35, further comprising fixing the orientation of the sound source toward the location in response to receiving a user input, such that the orientation of the sound source remains directed toward the location regardless of the location or the movement of the sound source.
41. The method of claim 35, further comprising: include: encoding the sound source into a bitstream for transmission to a playback device; as well as providing metadata including the set of parameters, The playback device is configured to render the sound source according to the set of parameters.
42. The method of claim 41, further comprising providing an updated set of parameters after providing the metadata, enabling the playback device to adjust the orientation of the sound source during rendering of the sound source.
43. An electronic device, the electronic device include: at least one processor; and a memory having instructions stored therein that, when executed by the at least one processor, cause the electronic device to: receiving an audio object of a sound program, the audio object comprising a radiation pattern for projecting sound in a three-dimensional (3D) scene toward a viewing direction of the audio object; assigning a point within the 3D scene at which the viewing direction of the audio object is to be directed; determining one or more orientation parameters for the look direction of the audio object; determining whether there is motion of the point or the audio object within the 3D scene; as well as In response to determining that there is motion of the point or the audio object, the one or more orientation parameters are updated based on the motion and without user intervention to maintain the viewing direction toward the point.
44. An electronic device according to claim 43, wherein the electronic device also includes a display, wherein the memory has additional instructions for a user interface (UI) for displaying the 3D scene on the display, and wherein the instructions for assigning the point include instructions for receiving a user selection of the point within the UI via an input device.
45. The electronic device of claim 44, wherein the audio object is a first audio object, wherein the point is one of: A static position relative to an origin within the 3D scene; a second audio object within the 3D scene; or A position within an area surrounded by a plurality of audio objects within the 3D scene.
46. The electronic device of claim 45, wherein the location is a centroid of the area.
47. An electronic device according to claim 43, wherein the sound program is a live audio broadcast.
48. The electronic device of claim 43, wherein the sound program is a live audio communication between the electronic device and a playback device.
49. A method, the method comprising: include: Decode the sound program's audio object from the bitstream; as well as Metadata of the sound program is received, wherein the metadata instructs the decoding side to lock the viewing direction of the radiation pattern of the audio object to be toward the point within a three-dimensional (3D) scene or the audio object when the point moves during playback of the sound program.
50. The method of claim 49, wherein the point is a location within the 3D scene, the metadata comprising coordinates of the location within a coordinate system relative to an origin of the 3D scene.
51. The method of claim 49, wherein the audio object is a first audio object, wherein the metadata comprises a reference to a second audio object, and the point is a position of the second audio object within the 3D scene.
52. The method of claim 49, wherein the metadata comprises a plurality of references to a plurality of audio objects within the 3D scene, wherein the method further include: determining positions of the plurality of audio objects based on the plurality of references; determining a position of the point within an area defined by the positions of the plurality of audio objects; as well as The look direction of the radiation pattern of the audio object from the location of the audio object to the location of the point within the 3D scene is determined.
53. The method of claim 49, wherein the point is located at a first position within the 3D scene, wherein the method further include: receiving additional metadata indicating that the point has moved to a second location within the 3D scene; as well as The viewing direction of the radiation pattern is moved from the first position to the second position over a period of time.
54. The method of claim 49, further comprising determining an angle between the first position and the second position relative to the position of the audio object within the 3D scene, wherein moving the viewing direction comprises rotating the viewing direction along the angle around the position of the audio object at a speed based on the time period.
55. The method of claim 49, wherein the metadata further comprises one or more angular offsets, wherein the method further comprises spatially rendering the audio object such that the viewing direction of the radiation pattern is offset from the point based on the one or more angular offsets.
56. The method of claim 49, further comprising: include: determining a position of a listener within the 3D scene; as well as generating a set of spatially rendered audio signals by spatially rendering the sound program according to the viewing direction of the radiation pattern of the audio object and the position of the listener; as well as A set of speakers is driven using the set of spatially rendered audio signals to play back the sound program.
57. A method according to claim 56, wherein the point within the 3D scene is the position of the listener, wherein the method further comprises determining one or more orientation parameters for orienting the viewing direction of the radiation pattern toward the position of the listener, wherein the set of spatially rendered audio signals is generated based on the one or more orientation parameters.
58. A method performed by an encoding side, the method include: encoding a sound program into a bitstream, the sound program comprising audio objects within a three-dimensional (3D) scene; as well as Metadata for the sound program is encoded into the bitstream, the metadata indicating a position within the 3D scene toward which a radiation pattern of the audio object will remain directed when the position or the audio object moves within the 3D scene during spatial audio rendering of the sound program.
59. The method of claim 58, wherein the metadata comprises a plurality of coordinates of the location, wherein the plurality of coordinates belongs to a coordinate system relative to an origin of the 3D scene.
60. The method of claim 58, wherein the audio object is a first audio object, wherein the metadata includes a reference to a second audio object at the location within the 3D scene such that when either audio object is rendered during spatial audio rendering of the sound program When moving within the 3D scene, the radiation pattern will remain directed towards the second audio object.
61. The method of claim 58, wherein the metadata comprises a plurality of references to a plurality of audio objects at different locations within the 3D scene, wherein the location is based on the plurality of locations.
62. The method of claim 61, wherein the position is a centroid of an area within the 3D scene defined by the positions of the plurality of audio objects.
63. A method according to claim 61, wherein encoding metadata includes adding audio object metadata, the audio object metadata including a reference to the metadata, the metadata instructing decoding-side processing to lock the direction of the radiation pattern to the position during spatial audio rendering of the sound program.
64. A method according to claim 61, wherein the metadata also includes one or more angular offsets, which cause the viewing direction of the radiation pattern to remain offset from the position when the position or the audio object moves within the 3D scene during spatial rendering of the sound program.
Citation Information
Cited By
Vehicle-mounted machine Dolby sound effect rendering method based on virtual machine and related equipment
CN121281530A
A virtual machine-based car-machine Dolby sound effect rendering method and related equipment
CN121281530B
Audio processing method, device and system, vehicle, storage medium and program product
CN122317506A
